[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:22:19 dist-test-slave-44d2 krb5kdc[16387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:22:19 dist-test-slave-44d2 krb5kdc[16387](info): set up 2 sockets
Aug 07 06:22:19 dist-test-slave-44d2 krb5kdc[16387](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260807 06:22:20.263197 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.262s user 0.007s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:22:20 dist-test-slave-44d2 krb5kdc[16387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:22:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:22:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:22:20.401047 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:36967
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:36967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:20.519593 16402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:20.519922 16402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:20.520005 16402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:20.524044 16402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:22:20.524137 16402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:20.524170 16402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:20.524210 16402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:22:20.524241 16402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:22:20.529582 16402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:36967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:36967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:20.530762 16402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:20.532003 16402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 06:22:20.541574 16402 server_base.cc:1034] running on GCE node
W20260807 06:22:20.541635 16410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:20.541646 16417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:20.541894 16412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:20.542582 16402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:20.543679 16402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:20.544848 16402 hybrid_clock.cc:648] HybridClock initialized: now 1786083740544821 us; error 40 us; skew 500 ppm
Aug 07 06:22:20 dist-test-slave-44d2 krb5kdc[16387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083740, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:20.547760 16402 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:22:20.548753 16402 webserver.cc:460] Webserver started at http://127.15.254.62:37709/ using document root <none> and password file <none>
I20260807 06:22:20.549302 16402 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:20.549362 16402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:20.549535 16402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:20.551661 16402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "95033ac57be842abbcb1ea68dca7010c"
format_stamp: "Formatted at 2026-08-07 06:22:20 on dist-test-slave-44d2"
server_key: "a3dc0b59ab139a054130dce37c92a37a"
server_key_iv: "83502d0aa32a7d7da2677d85149bab26"
server_key_version: "encryptionkey@0"
I20260807 06:22:20.552211 16402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "95033ac57be842abbcb1ea68dca7010c"
format_stamp: "Formatted at 2026-08-07 06:22:20 on dist-test-slave-44d2"
server_key: "a3dc0b59ab139a054130dce37c92a37a"
server_key_iv: "83502d0aa32a7d7da2677d85149bab26"
server_key_version: "encryptionkey@0"
I20260807 06:22:20.556120 16402 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 06:22:20.558990 16427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:20.560282 16402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:22:20.560427 16402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "95033ac57be842abbcb1ea68dca7010c"
format_stamp: "Formatted at 2026-08-07 06:22:20 on dist-test-slave-44d2"
server_key: "a3dc0b59ab139a054130dce37c92a37a"
server_key_iv: "83502d0aa32a7d7da2677d85149bab26"
server_key_version: "encryptionkey@0"
I20260807 06:22:20.560598 16402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:20.584072 16402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:20.587877 16402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:20.588128 16402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:20.597189 16402 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:36967
I20260807 06:22:20.597332 16505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:36967 every 8 connection(s)
I20260807 06:22:20.598392 16402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:22:20.601629 16507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:20.607719 16507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95033ac57be842abbcb1ea68dca7010c: Bootstrap starting.
I20260807 06:22:20.609129 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 16402
I20260807 06:22:20.609290 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:22:20.610637 16507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95033ac57be842abbcb1ea68dca7010c: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:20.610678 16376 external_mini_cluster.cc:1428] Setting key 89f621738139b02f6b1af6c956b88950
I20260807 06:22:20.611577 16507 log.cc:824] T 00000000000000000000000000000000 P 95033ac57be842abbcb1ea68dca7010c: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:20.614040 16507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95033ac57be842abbcb1ea68dca7010c: No bootstrap required, opened a new log
I20260807 06:22:20.618716 16507 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95033ac57be842abbcb1ea68dca7010c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95033ac57be842abbcb1ea68dca7010c" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36967 } }
I20260807 06:22:20.618991 16507 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95033ac57be842abbcb1ea68dca7010c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:20.619056 16507 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95033ac57be842abbcb1ea68dca7010c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95033ac57be842abbcb1ea68dca7010c, State: Initialized, Role: FOLLOWER
I20260807 06:22:20.619575 16507 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95033ac57be842abbcb1ea68dca7010c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95033ac57be842abbcb1ea68dca7010c" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36967 } }
I20260807 06:22:20.619741 16507 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95033ac57be842abbcb1ea68dca7010c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:22:20.619822 16507 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95033ac57be842abbcb1ea68dca7010c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:22:20.619979 16507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95033ac57be842abbcb1ea68dca7010c [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:20.621016 16507 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95033ac57be842abbcb1ea68dca7010c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95033ac57be842abbcb1ea68dca7010c" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36967 } }
I20260807 06:22:20.621359 16507 leader_election.cc:302] T 00000000000000000000000000000000 P 95033ac57be842abbcb1ea68dca7010c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95033ac57be842abbcb1ea68dca7010c; no voters:
I20260807 06:22:20.621685 16507 leader_election.cc:288] T 00000000000000000000000000000000 P 95033ac57be842abbcb1ea68dca7010c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:22:20.621747 16514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95033ac57be842abbcb1ea68dca7010c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:20.621975 16514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95033ac57be842abbcb1ea68dca7010c [term 1 LEADER]: Becoming Leader. State: Replica: 95033ac57be842abbcb1ea68dca7010c, State: Running, Role: LEADER
I20260807 06:22:20.622287 16514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95033ac57be842abbcb1ea68dca7010c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95033ac57be842abbcb1ea68dca7010c" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36967 } }
I20260807 06:22:20.623034 16507 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95033ac57be842abbcb1ea68dca7010c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:22:20.623504 16514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95033ac57be842abbcb1ea68dca7010c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95033ac57be842abbcb1ea68dca7010c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95033ac57be842abbcb1ea68dca7010c" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36967 } } }
I20260807 06:22:20.623497 16517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95033ac57be842abbcb1ea68dca7010c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95033ac57be842abbcb1ea68dca7010c. Latest consensus state: current_term: 1 leader_uuid: "95033ac57be842abbcb1ea68dca7010c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95033ac57be842abbcb1ea68dca7010c" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36967 } } }
I20260807 06:22:20.623772 16514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95033ac57be842abbcb1ea68dca7010c [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:20.623786 16517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95033ac57be842abbcb1ea68dca7010c [sys.catalog]: This master's current role is: LEADER
Aug 07 06:22:20 dist-test-slave-44d2 krb5kdc[16387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
W20260807 06:22:20.630434 16539 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 95033ac57be842abbcb1ea68dca7010c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:22:20.630520 16539 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:22:20.630599 16525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:22:20.631451 16525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:22:20.637626 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:20.614083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40220 (local address 127.15.254.62:36967)
0807 06:22:20.614624 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:20.614645 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:20.615926 (+ 1281us) server_negotiation.cc:407] Connection header received
0807 06:22:20.618721 (+ 2795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:20.618740 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:20.619153 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:20.619517 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:20.621977 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:20.623094 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:20.625105 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:20.625937 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:20.626141 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:20.631070 (+ 4929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:20.631076 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:20.631089 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:20.631116 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:20.633374 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:20.634029 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:20.634035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:20.634042 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:20.634206 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:20.634672 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:20.634677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:20.634679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:20.635284 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:20.635547 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:20.636225 (+ 678us) server_negotiation.cc:299] Negotiation successful
0807 06:22:20.636557 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":151,"threads_started":1}
I20260807 06:22:20.638761 16525 catalog_manager.cc:1347] Generated new cluster ID: 21b1398c2940401883948046db5c5c13
I20260807 06:22:20.638913 16525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:22:20.651314 16525 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:22:20.653306 16525 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:22:20.670925 16525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95033ac57be842abbcb1ea68dca7010c: Generated new TSK 0
I20260807 06:22:20.672163 16525 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
I20260807 06:22:20.720340 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:0
--local_ip_for_outbound_sockets=127.15.254.1
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:36967
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:20.877243 16546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:20.877513 16546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:20.877559 16546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:20.882242 16546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:20.882310 16546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:20.882377 16546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:22:20.887502 16546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:36967
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:20.888669 16546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:20.889732 16546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:20.900913 16552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:20.900944 16551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:20.901081 16546 server_base.cc:1034] running on GCE node
W20260807 06:22:20.900888 16554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:20.901589 16546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:20.902281 16546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:20.903461 16546 hybrid_clock.cc:648] HybridClock initialized: now 1786083740903444 us; error 35 us; skew 500 ppm
Aug 07 06:22:20 dist-test-slave-44d2 krb5kdc[16387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083740, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:20.906534 16546 init.cc:377] Logged in from keytab as kudu/127.15.254.1@KRBTEST.COM (short username kudu)
I20260807 06:22:20.907431 16546 webserver.cc:460] Webserver started at http://127.15.254.1:42395/ using document root <none> and password file <none>
I20260807 06:22:20.907968 16546 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:20.908028 16546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:20.908197 16546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:20.910355 16546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-0/data/instance:
uuid: "9cdfbdcea38f43b18f785c675505c796"
format_stamp: "Formatted at 2026-08-07 06:22:20 on dist-test-slave-44d2"
server_key: "87f149ec7fa762d026b9fec52b82d075"
server_key_iv: "1d043f7839046d15be75c2ca4485c416"
server_key_version: "encryptionkey@0"
I20260807 06:22:20.910908 16546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance:
uuid: "9cdfbdcea38f43b18f785c675505c796"
format_stamp: "Formatted at 2026-08-07 06:22:20 on dist-test-slave-44d2"
server_key: "87f149ec7fa762d026b9fec52b82d075"
server_key_iv: "1d043f7839046d15be75c2ca4485c416"
server_key_version: "encryptionkey@0"
I20260807 06:22:20.914867 16546 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:22:20.917518 16563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:20.919020 16546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:22:20.919188 16546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "9cdfbdcea38f43b18f785c675505c796"
format_stamp: "Formatted at 2026-08-07 06:22:20 on dist-test-slave-44d2"
server_key: "87f149ec7fa762d026b9fec52b82d075"
server_key_iv: "1d043f7839046d15be75c2ca4485c416"
server_key_version: "encryptionkey@0"
I20260807 06:22:20.919315 16546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:20.938072 16546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:20.941895 16546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:20.942128 16546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:20.942842 16546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:20.943786 16546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:20.943862 16546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:20.943950 16546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:20.944006 16546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:20.955523 16546 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:43813
I20260807 06:22:20.955550 16723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:43813 every 8 connection(s)
I20260807 06:22:20.956552 16546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:22:20.959762 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 16546
I20260807 06:22:20.959887 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance
I20260807 06:22:20.960209 16376 external_mini_cluster.cc:1428] Setting key addb63c6558d48fa0c93d4ef01a8fa5f
Aug 07 06:22:20 dist-test-slave-44d2 krb5kdc[16387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083740, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.254.2@KRBTEST.COM; defaulting to no policy
I20260807 06:22:20.973780 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:20.958710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:41017 (local address 127.15.254.62:36967)
0807 06:22:20.958856 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:20.958860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:20.960300 (+ 1440us) server_negotiation.cc:407] Connection header received
0807 06:22:20.961446 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:20.961450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:20.961501 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:20.961587 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:20.963315 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:20.963916 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:20.965467 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:20.965923 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:20.966069 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:20.969017 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:20.969021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:20.969023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:20.969050 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:20.970836 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:20.971610 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:20.971613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:20.971615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:20.971718 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:20.972223 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:20.972226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:20.972228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:20.972459 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:20.972639 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:20.973429 (+ 790us) server_negotiation.cc:299] Negotiation successful
0807 06:22:20.973599 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 06:22:20.974787 16727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:20.959144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:36967 (local address 127.15.254.1:41017)
0807 06:22:20.960065 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:20.960134 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:20.961186 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:20.961816 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:20.961824 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:20.962361 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:20.963136 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:20.963160 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:20.964110 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:20.964117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:20.965317 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:20.965331 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:20.966132 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:20.966136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:20.966322 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:20.967125 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:20.967143 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:20.968816 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:20.971004 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:20.971015 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:20.971039 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:20.971479 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:20.971828 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:20.971833 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:20.971838 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:20.972074 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:20.972656 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:20.972666 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:20.973090 (+ 424us) client_negotiation.cc:769] Sending connection context
0807 06:22:20.973332 (+ 242us) client_negotiation.cc:240] Negotiation successful
0807 06:22:20.973780 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":137,"threads_started":1}
I20260807 06:22:20.976171 16724 heartbeater.cc:347] Connected to a master server at 127.15.254.62:36967
I20260807 06:22:20.976442 16724 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.2@KRBTEST.COM" created.
I20260807 06:22:20.977069 16724 heartbeater.cc:511] Master 127.15.254.62:36967 requested a full tablet report, sending...
I20260807 06:22:20.979015 16449 ts_manager.cc:194] Registered new tserver with Master: 9cdfbdcea38f43b18f785c675505c796 (127.15.254.1:43813)
I20260807 06:22:20.980228 16449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.1@KRBTEST.COM'} at 127.15.254.1:41017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
I20260807 06:22:20.995885 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:0
--local_ip_for_outbound_sockets=127.15.254.2
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:36967
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:21.120620 16730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:21.120888 16730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:21.120965 16730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:21.125263 16730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:21.125345 16730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:21.125491 16730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:22:21.130827 16730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:36967
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:21.131997 16730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:21.133167 16730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:21.143919 16741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:21.144014 16738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:21.143953 16739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:21.144095 16730 server_base.cc:1034] running on GCE node
I20260807 06:22:21.144615 16730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:21.145299 16730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:21.146466 16730 hybrid_clock.cc:648] HybridClock initialized: now 1786083741146446 us; error 36 us; skew 500 ppm
Aug 07 06:22:21 dist-test-slave-44d2 krb5kdc[16387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083741, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:21.149525 16730 init.cc:377] Logged in from keytab as kudu/127.15.254.2@KRBTEST.COM (short username kudu)
I20260807 06:22:21.150415 16730 webserver.cc:460] Webserver started at http://127.15.254.2:36709/ using document root <none> and password file <none>
I20260807 06:22:21.150981 16730 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:21.151047 16730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:21.151289 16730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:21.153436 16730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-1/data/instance:
uuid: "53fac97ed9054a01a52ec4e005dffcc2"
format_stamp: "Formatted at 2026-08-07 06:22:21 on dist-test-slave-44d2"
server_key: "717d50dd7d3ffc6481b37937c0b7c8ce"
server_key_iv: "4977300e1065c12eeb8a47f2ca7a2f56"
server_key_version: "encryptionkey@0"
I20260807 06:22:21.154028 16730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance:
uuid: "53fac97ed9054a01a52ec4e005dffcc2"
format_stamp: "Formatted at 2026-08-07 06:22:21 on dist-test-slave-44d2"
server_key: "717d50dd7d3ffc6481b37937c0b7c8ce"
server_key_iv: "4977300e1065c12eeb8a47f2ca7a2f56"
server_key_version: "encryptionkey@0"
I20260807 06:22:21.157809 16730 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:22:21.160249 16750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:21.161790 16730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 06:22:21.161916 16730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "53fac97ed9054a01a52ec4e005dffcc2"
format_stamp: "Formatted at 2026-08-07 06:22:21 on dist-test-slave-44d2"
server_key: "717d50dd7d3ffc6481b37937c0b7c8ce"
server_key_iv: "4977300e1065c12eeb8a47f2ca7a2f56"
server_key_version: "encryptionkey@0"
I20260807 06:22:21.162042 16730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:21.176445 16730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:21.180014 16730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:21.180246 16730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:21.180994 16730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:21.181883 16730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:21.181931 16730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:21.182003 16730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:21.182046 16730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:21.194026 16730 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:39933
I20260807 06:22:21.194069 16907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:39933 every 8 connection(s)
I20260807 06:22:21.194968 16730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:22:21.203661 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 16730
I20260807 06:22:21.203788 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance
I20260807 06:22:21.204097 16376 external_mini_cluster.cc:1428] Setting key 5b577af75715d64eab99531dea9de2e4
Aug 07 06:22:21 dist-test-slave-44d2 krb5kdc[16387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083741, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:21.209987 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:21.197040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:44257 (local address 127.15.254.62:36967)
0807 06:22:21.197215 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:21.197220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:21.198340 (+ 1120us) server_negotiation.cc:407] Connection header received
0807 06:22:21.199374 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:21.199377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:21.199439 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:21.199523 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:21.200836 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:21.201381 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:21.202593 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:21.202809 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:21.202890 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:21.205759 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:21.205763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:21.205766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:21.205795 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:21.207586 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:21.208328 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:21.208332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:21.208335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:21.208433 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:21.208836 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:21.208839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:21.208841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:21.209032 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:21.209145 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:21.209658 (+ 513us) server_negotiation.cc:299] Negotiation successful
0807 06:22:21.209798 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 06:22:21.210829 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:21.197482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:36967 (local address 127.15.254.2:44257)
0807 06:22:21.198188 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:21.198231 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:21.199114 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:21.199696 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:21.199703 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:21.200093 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:21.200689 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:21.200706 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:21.201515 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:21.201518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:21.202480 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:21.202490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:21.202944 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:21.202946 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:21.203098 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:21.203715 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:21.203741 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:21.205575 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:21.207722 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:21.207731 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:21.207748 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:21.208188 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:21.208584 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:21.208586 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:21.208587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:21.208721 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:21.209144 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:21.209152 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:21.209402 (+ 250us) client_negotiation.cc:769] Sending connection context
0807 06:22:21.209598 (+ 196us) client_negotiation.cc:240] Negotiation successful
0807 06:22:21.209950 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":119,"threads_started":1}
I20260807 06:22:21.212155 16908 heartbeater.cc:347] Connected to a master server at 127.15.254.62:36967
I20260807 06:22:21.212435 16908 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:21.212986 16908 heartbeater.cc:511] Master 127.15.254.62:36967 requested a full tablet report, sending...
I20260807 06:22:21.214135 16449 ts_manager.cc:194] Registered new tserver with Master: 53fac97ed9054a01a52ec4e005dffcc2 (127.15.254.2:39933)
I20260807 06:22:21.214980 16449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.2@KRBTEST.COM'} at 127.15.254.2:44257
WARNING: no policy specified for kudu/127.15.254.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
I20260807 06:22:21.236101 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:0
--local_ip_for_outbound_sockets=127.15.254.3
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:36967
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:21.364388 16917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:21.364750 16917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:21.364838 16917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:21.369109 16917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:21.369168 16917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:21.369272 16917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:22:21.374832 16917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:36967
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:21.376133 16917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:21.377323 16917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 06:22:21.387810 16917 server_base.cc:1034] running on GCE node
W20260807 06:22:21.387810 16923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:21.387813 16924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:21.387823 16926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:21.388626 16917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:21.389287 16917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:21.390448 16917 hybrid_clock.cc:648] HybridClock initialized: now 1786083741390431 us; error 30 us; skew 500 ppm
Aug 07 06:22:21 dist-test-slave-44d2 krb5kdc[16387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083741, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:21.393406 16917 init.cc:377] Logged in from keytab as kudu/127.15.254.3@KRBTEST.COM (short username kudu)
I20260807 06:22:21.394258 16917 webserver.cc:460] Webserver started at http://127.15.254.3:38337/ using document root <none> and password file <none>
I20260807 06:22:21.394822 16917 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:21.394884 16917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:21.395124 16917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:21.397291 16917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-2/data/instance:
uuid: "72421099ce744e7d9585be2e838ac894"
format_stamp: "Formatted at 2026-08-07 06:22:21 on dist-test-slave-44d2"
server_key: "9f1dbfc599d53f8562189f5060f3f870"
server_key_iv: "11a1ef775fa28127061a072fbd2cff39"
server_key_version: "encryptionkey@0"
I20260807 06:22:21.397864 16917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance:
uuid: "72421099ce744e7d9585be2e838ac894"
format_stamp: "Formatted at 2026-08-07 06:22:21 on dist-test-slave-44d2"
server_key: "9f1dbfc599d53f8562189f5060f3f870"
server_key_iv: "11a1ef775fa28127061a072fbd2cff39"
server_key_version: "encryptionkey@0"
I20260807 06:22:21.401816 16917 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:22:21.404456 16935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:21.405771 16917 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 06:22:21.405911 16917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "72421099ce744e7d9585be2e838ac894"
format_stamp: "Formatted at 2026-08-07 06:22:21 on dist-test-slave-44d2"
server_key: "9f1dbfc599d53f8562189f5060f3f870"
server_key_iv: "11a1ef775fa28127061a072fbd2cff39"
server_key_version: "encryptionkey@0"
I20260807 06:22:21.406057 16917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:21.421126 16917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:21.424913 16917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:21.425150 16917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:21.425812 16917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:21.426694 16917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:21.426769 16917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:21.426842 16917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:21.426884 16917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:21.438694 16917 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:41269
I20260807 06:22:21.438724 17096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:41269 every 8 connection(s)
I20260807 06:22:21.439647 16917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:22:21.443423 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 16917
I20260807 06:22:21.443563 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance
I20260807 06:22:21.443831 16376 external_mini_cluster.cc:1428] Setting key b53795efb3ff15af4832b57a4ad9d25a
Aug 07 06:22:21 dist-test-slave-44d2 krb5kdc[16387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083741, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:21.454130 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:21.441617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:54259 (local address 127.15.254.62:36967)
0807 06:22:21.441761 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:21.441765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:21.442873 (+ 1108us) server_negotiation.cc:407] Connection header received
0807 06:22:21.443974 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:21.443978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:21.444047 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:21.444146 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:21.445620 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:21.446328 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:21.447539 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:21.447759 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:21.447872 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:21.450460 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:21.450464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:21.450467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:21.450502 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:21.451804 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:21.452367 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:21.452370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:21.452372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:21.452459 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:21.452901 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:21.452906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:21.452910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:21.453119 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:21.453238 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:21.453797 (+ 559us) server_negotiation.cc:299] Negotiation successful
0807 06:22:21.453933 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 06:22:21.455001 17102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:21.442034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:36967 (local address 127.15.254.3:54259)
0807 06:22:21.442722 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:21.442766 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:21.443737 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:21.444310 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:21.444336 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:21.444814 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:21.445491 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:21.445509 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:21.446457 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:21.446461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:21.447388 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:21.447399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:21.447911 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:21.447916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:21.448094 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:21.448744 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:21.448760 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:21.450289 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:21.451933 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:21.451939 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:21.451956 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:21.452245 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:21.452640 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:21.452644 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:21.452647 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:21.452799 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:21.453247 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:21.453257 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:21.453533 (+ 276us) client_negotiation.cc:769] Sending connection context
0807 06:22:21.453738 (+ 205us) client_negotiation.cc:240] Negotiation successful
0807 06:22:21.454134 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":110,"threads_started":1}
I20260807 06:22:21.456259 17098 heartbeater.cc:347] Connected to a master server at 127.15.254.62:36967
I20260807 06:22:21.456545 17098 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:21.457207 17098 heartbeater.cc:511] Master 127.15.254.62:36967 requested a full tablet report, sending...
I20260807 06:22:21.458469 16448 ts_manager.cc:194] Registered new tserver with Master: 72421099ce744e7d9585be2e838ac894 (127.15.254.3:41269)
I20260807 06:22:21.459100 16448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.3@KRBTEST.COM'} at 127.15.254.3:54259
I20260807 06:22:21.470551 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 06:22:21 dist-test-slave-44d2 krb5kdc[16387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083741, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 06:22:21 dist-test-slave-44d2 krb5kdc[16387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083741, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.254.1@KRBTEST.COM
I20260807 06:22:21.499742 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:21.489971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50860 (local address 127.15.254.1:43813)
0807 06:22:21.490349 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:21.490356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:21.490381 (+ 25us) server_negotiation.cc:407] Connection header received
0807 06:22:21.490437 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:21.490442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:21.490663 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:21.490793 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:21.491537 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:21.492318 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:21.493236 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:21.493640 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:21.493727 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:21.496192 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:21.496203 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:21.496218 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:21.496243 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:21.498069 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:21.498466 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:21.498471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:21.498472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:21.498555 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:21.498857 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:21.498859 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:21.498860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:21.499113 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:21.499274 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:21.499451 (+ 177us) server_negotiation.cc:299] Negotiation successful
0807 06:22:21.499593 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":134,"threads_started":1}
W20260807 06:22:21.500854 16600 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50860
Aug 07 06:22:21 dist-test-slave-44d2 krb5kdc[16387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 06:22:21 dist-test-slave-44d2 krb5kdc[16387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.1@KRBTEST.COM
I20260807 06:22:21.529829 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:21.520742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50864 (local address 127.15.254.1:43813)
0807 06:22:21.520898 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:21.520903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:21.521074 (+ 171us) server_negotiation.cc:407] Connection header received
0807 06:22:21.521269 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:21.521274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:21.521331 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:21.521422 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:21.522171 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:21.522707 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:21.523731 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:21.523976 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:21.524089 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:21.526815 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:21.526818 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:21.526820 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:21.526850 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:21.528147 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:21.528613 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:21.528616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:21.528617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:21.528701 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:21.528974 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:21.528976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:21.528978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:21.529207 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:21.529334 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:21.529544 (+ 210us) server_negotiation.cc:299] Negotiation successful
0807 06:22:21.529675 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 06:22:21.531250 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 16546
I20260807 06:22:21.547009 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 16730
I20260807 06:22:21.555234 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 16917
I20260807 06:22:21.565218 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 16402
2026-08-07T06:22:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2604 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:22:21 dist-test-slave-44d2 krb5kdc[17123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:22:21 dist-test-slave-44d2 krb5kdc[17123](info): set up 2 sockets
Aug 07 06:22:21 dist-test-slave-44d2 krb5kdc[17123](info): commencing operation
krb5kdc: starting...
W20260807 06:22:22.820403 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.225s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:22:22 dist-test-slave-44d2 krb5kdc[17123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:22:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:22:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:22:22.961843 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:43029
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:44331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:43029
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:23.079115 17145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:23.079449 17145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:23.079530 17145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:23.083585 17145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:22:23.083707 17145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:23.083742 17145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:23.083781 17145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:22:23.083806 17145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:22:23.089188 17145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:44331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:43029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:43029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:23.090346 17145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:23.091507 17145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:23.101073 17157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:23.101110 17154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:23.101073 17155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:23.101270 17145 server_base.cc:1034] running on GCE node
I20260807 06:22:23.102049 17145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:23.103247 17145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:23.104449 17145 hybrid_clock.cc:648] HybridClock initialized: now 1786083743104415 us; error 43 us; skew 500 ppm
Aug 07 06:22:23 dist-test-slave-44d2 krb5kdc[17123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083743, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:23.107595 17145 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:22:23.108565 17145 webserver.cc:460] Webserver started at http://127.15.254.62:43453/ using document root <none> and password file <none>
I20260807 06:22:23.109167 17145 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:23.109230 17145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:23.109486 17145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:23.111634 17145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "93e34ce2268145c8899fda7b0747347c"
format_stamp: "Formatted at 2026-08-07 06:22:23 on dist-test-slave-44d2"
server_key: "f1976186eda6475c73df1fd64da6bc74"
server_key_iv: "a1d35191f565a49ce1b4eb849ab8e1a6"
server_key_version: "encryptionkey@0"
I20260807 06:22:23.112221 17145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "93e34ce2268145c8899fda7b0747347c"
format_stamp: "Formatted at 2026-08-07 06:22:23 on dist-test-slave-44d2"
server_key: "f1976186eda6475c73df1fd64da6bc74"
server_key_iv: "a1d35191f565a49ce1b4eb849ab8e1a6"
server_key_version: "encryptionkey@0"
I20260807 06:22:23.116067 17145 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:22:23.118675 17164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:23.120179 17145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 06:22:23.120337 17145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "93e34ce2268145c8899fda7b0747347c"
format_stamp: "Formatted at 2026-08-07 06:22:23 on dist-test-slave-44d2"
server_key: "f1976186eda6475c73df1fd64da6bc74"
server_key_iv: "a1d35191f565a49ce1b4eb849ab8e1a6"
server_key_version: "encryptionkey@0"
I20260807 06:22:23.120452 17145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:23.141386 17145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:23.145255 17145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:23.145500 17145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:23.153199 17145 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:43029
I20260807 06:22:23.153230 17249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:43029 every 8 connection(s)
I20260807 06:22:23.154204 17145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:22:23.157189 17250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:23.159797 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 17145
I20260807 06:22:23.159917 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:22:23.160204 16376 external_mini_cluster.cc:1428] Setting key dbbd4bacc78c6d7659f535fc678c965e
I20260807 06:22:23.163753 17250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93e34ce2268145c8899fda7b0747347c: Bootstrap starting.
I20260807 06:22:23.166127 17250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93e34ce2268145c8899fda7b0747347c: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:23.166922 17250 log.cc:824] T 00000000000000000000000000000000 P 93e34ce2268145c8899fda7b0747347c: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:23.168953 17250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93e34ce2268145c8899fda7b0747347c: No bootstrap required, opened a new log
Aug 07 06:22:23 dist-test-slave-44d2 krb5kdc[17123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:23.172642 17250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93e34ce2268145c8899fda7b0747347c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e34ce2268145c8899fda7b0747347c" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43029 } }
I20260807 06:22:23.172933 17250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93e34ce2268145c8899fda7b0747347c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:23.173022 17250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93e34ce2268145c8899fda7b0747347c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e34ce2268145c8899fda7b0747347c, State: Initialized, Role: FOLLOWER
I20260807 06:22:23.173537 17250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93e34ce2268145c8899fda7b0747347c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e34ce2268145c8899fda7b0747347c" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43029 } }
I20260807 06:22:23.173712 17250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93e34ce2268145c8899fda7b0747347c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:22:23.173791 17250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93e34ce2268145c8899fda7b0747347c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:22:23.173907 17250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93e34ce2268145c8899fda7b0747347c [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:23.174849 17255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:23.161776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46924 (local address 127.15.254.62:43029)
0807 06:22:23.162238 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:23.162253 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:23.162291 (+ 38us) server_negotiation.cc:407] Connection header received
0807 06:22:23.162933 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:23.162949 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:23.163301 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:23.163662 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:23.164520 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:23.165541 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:23.166462 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:23.166946 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:23.167135 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:23.169791 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:23.169798 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:23.169817 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:23.169843 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:23.171644 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:23.172120 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:23.172131 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:23.172136 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:23.172252 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:23.172617 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:23.172622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:23.172624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:23.173174 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:23.173414 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:23.173704 (+ 290us) server_negotiation.cc:299] Negotiation successful
0807 06:22:23.174085 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":132,"threads_started":1}
I20260807 06:22:23.174929 17250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93e34ce2268145c8899fda7b0747347c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e34ce2268145c8899fda7b0747347c" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43029 } }
I20260807 06:22:23.175292 17250 leader_election.cc:302] T 00000000000000000000000000000000 P 93e34ce2268145c8899fda7b0747347c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93e34ce2268145c8899fda7b0747347c; no voters:
I20260807 06:22:23.175678 17250 leader_election.cc:288] T 00000000000000000000000000000000 P 93e34ce2268145c8899fda7b0747347c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:22:23.175765 17257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93e34ce2268145c8899fda7b0747347c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:23.175976 17257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93e34ce2268145c8899fda7b0747347c [term 1 LEADER]: Becoming Leader. State: Replica: 93e34ce2268145c8899fda7b0747347c, State: Running, Role: LEADER
I20260807 06:22:23.176307 17257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93e34ce2268145c8899fda7b0747347c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e34ce2268145c8899fda7b0747347c" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43029 } }
I20260807 06:22:23.177098 17250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93e34ce2268145c8899fda7b0747347c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:22:23.177565 17257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93e34ce2268145c8899fda7b0747347c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93e34ce2268145c8899fda7b0747347c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e34ce2268145c8899fda7b0747347c" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43029 } } }
I20260807 06:22:23.177688 17257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93e34ce2268145c8899fda7b0747347c [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:23.177911 17257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93e34ce2268145c8899fda7b0747347c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93e34ce2268145c8899fda7b0747347c. Latest consensus state: current_term: 1 leader_uuid: "93e34ce2268145c8899fda7b0747347c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e34ce2268145c8899fda7b0747347c" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43029 } } }
I20260807 06:22:23.178014 17257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93e34ce2268145c8899fda7b0747347c [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:23.178130 17261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:22:23.181354 17261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:22:23.187397 17261 catalog_manager.cc:1347] Generated new cluster ID: 1009473a5df44b1e88d4951c0d20b8d9
I20260807 06:22:23.187469 17261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:22:23.202078 17261 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:22:23.203122 17261 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:22:23.211354 17261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93e34ce2268145c8899fda7b0747347c: Generated new TSK 0
I20260807 06:22:23.212338 17261 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
I20260807 06:22:23.248663 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:0
--local_ip_for_outbound_sockets=127.15.254.1
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:43029
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:44331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:23.372370 17286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:23.372813 17286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:23.372913 17286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:23.377123 17286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:23.377240 17286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:23.377374 17286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:22:23.383293 17286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:44331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:43029
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:23.384470 17286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:23.385582 17286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:23.395723 17296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:23.395730 17293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:23.395730 17294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:23.396203 17286 server_base.cc:1034] running on GCE node
I20260807 06:22:23.396661 17286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:23.397396 17286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:23.398594 17286 hybrid_clock.cc:648] HybridClock initialized: now 1786083743398577 us; error 33 us; skew 500 ppm
Aug 07 06:22:23 dist-test-slave-44d2 krb5kdc[17123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083743, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:23.401484 17286 init.cc:377] Logged in from keytab as kudu/127.15.254.1@KRBTEST.COM (short username kudu)
I20260807 06:22:23.402294 17286 webserver.cc:460] Webserver started at http://127.15.254.1:38949/ using document root <none> and password file <none>
I20260807 06:22:23.402820 17286 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:23.402878 17286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:23.403055 17286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:23.405207 17286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-0/data/instance:
uuid: "45c9564409b14caabc2141c8a52ff4b3"
format_stamp: "Formatted at 2026-08-07 06:22:23 on dist-test-slave-44d2"
server_key: "5aeda2e039120b136d6e2647954fc070"
server_key_iv: "4da9b6939033b9da3265761d9f5fd214"
server_key_version: "encryptionkey@0"
I20260807 06:22:23.405737 17286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance:
uuid: "45c9564409b14caabc2141c8a52ff4b3"
format_stamp: "Formatted at 2026-08-07 06:22:23 on dist-test-slave-44d2"
server_key: "5aeda2e039120b136d6e2647954fc070"
server_key_iv: "4da9b6939033b9da3265761d9f5fd214"
server_key_version: "encryptionkey@0"
I20260807 06:22:23.409595 17286 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 06:22:23.412223 17306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:23.413782 17286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:22:23.413908 17286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "45c9564409b14caabc2141c8a52ff4b3"
format_stamp: "Formatted at 2026-08-07 06:22:23 on dist-test-slave-44d2"
server_key: "5aeda2e039120b136d6e2647954fc070"
server_key_iv: "4da9b6939033b9da3265761d9f5fd214"
server_key_version: "encryptionkey@0"
I20260807 06:22:23.414017 17286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:23.441078 17286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:23.444691 17286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:23.444869 17286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:23.445468 17286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:23.446329 17286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:23.446376 17286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:23.446419 17286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:23.446434 17286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:23.456568 17286 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:35853
I20260807 06:22:23.456674 17468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:35853 every 8 connection(s)
I20260807 06:22:23.457636 17286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:22:23.467519 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 17286
I20260807 06:22:23.467665 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance
I20260807 06:22:23.467993 16376 external_mini_cluster.cc:1428] Setting key 70c788ca1338213947440c6dbf65ea5a
Aug 07 06:22:23 dist-test-slave-44d2 krb5kdc[17123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083743, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:23.473964 17255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:23.459834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:53465 (local address 127.15.254.62:43029)
0807 06:22:23.459997 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:23.460002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:23.460940 (+ 938us) server_negotiation.cc:407] Connection header received
0807 06:22:23.462344 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:23.462347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:23.462402 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:23.462495 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:23.463968 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:23.464663 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:23.465909 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:23.466188 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:23.466277 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:23.469430 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:23.469433 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:23.469436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:23.469490 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:23.471120 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:23.471972 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:23.471978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:23.471981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:23.472092 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:23.472467 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:23.472472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:23.472476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:23.472804 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:23.472944 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:23.473532 (+ 588us) server_negotiation.cc:299] Negotiation successful
0807 06:22:23.473712 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 06:22:23.474697 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:23.460170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:43029 (local address 127.15.254.1:53465)
0807 06:22:23.460766 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:23.460818 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:23.462079 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:23.462715 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:23.462722 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:23.463199 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:23.463729 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:23.463743 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:23.464816 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:23.464819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:23.465785 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:23.465796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:23.466359 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:23.466362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:23.466522 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:23.467221 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:23.467248 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:23.469249 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:23.471257 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:23.471267 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:23.471288 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:23.471699 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:23.472205 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:23.472207 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:23.472208 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:23.472363 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:23.472956 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:23.472962 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:23.473264 (+ 302us) client_negotiation.cc:769] Sending connection context
0807 06:22:23.473436 (+ 172us) client_negotiation.cc:240] Negotiation successful
0807 06:22:23.473719 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":124,"threads_started":1}
I20260807 06:22:23.476155 17469 heartbeater.cc:347] Connected to a master server at 127.15.254.62:43029
I20260807 06:22:23.476502 17469 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:23.477293 17469 heartbeater.cc:511] Master 127.15.254.62:43029 requested a full tablet report, sending...
I20260807 06:22:23.479524 17196 ts_manager.cc:194] Registered new tserver with Master: 45c9564409b14caabc2141c8a52ff4b3 (127.15.254.1:35853)
WARNING: no policy specified for kudu/127.15.254.2@KRBTEST.COM; defaulting to no policy
I20260807 06:22:23.480906 17196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.1@KRBTEST.COM'} at 127.15.254.1:53465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
I20260807 06:22:23.504417 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:0
--local_ip_for_outbound_sockets=127.15.254.2
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:43029
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:44331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:23.622838 17475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:23.623152 17475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:23.623229 17475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:23.627449 17475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:23.627549 17475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:23.627648 17475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:22:23.633407 17475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:44331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:43029
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:23.634509 17475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:23.635560 17475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:23.645569 17481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:23.645601 17482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:23.645744 17484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:23.645777 17475 server_base.cc:1034] running on GCE node
I20260807 06:22:23.646319 17475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:23.647053 17475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:23.648260 17475 hybrid_clock.cc:648] HybridClock initialized: now 1786083743648230 us; error 52 us; skew 500 ppm
Aug 07 06:22:23 dist-test-slave-44d2 krb5kdc[17123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083743, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:23.651438 17475 init.cc:377] Logged in from keytab as kudu/127.15.254.2@KRBTEST.COM (short username kudu)
I20260807 06:22:23.652438 17475 webserver.cc:460] Webserver started at http://127.15.254.2:35241/ using document root <none> and password file <none>
I20260807 06:22:23.653172 17475 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:23.653285 17475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:23.653548 17475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:23.655758 17475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-1/data/instance:
uuid: "647a80a9fa0146c2b5abe8f23f2e803b"
format_stamp: "Formatted at 2026-08-07 06:22:23 on dist-test-slave-44d2"
server_key: "a1a567018d1e268f559ec38bde54b517"
server_key_iv: "d4067584a7da1019f52b42001d9431aa"
server_key_version: "encryptionkey@0"
I20260807 06:22:23.656379 17475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance:
uuid: "647a80a9fa0146c2b5abe8f23f2e803b"
format_stamp: "Formatted at 2026-08-07 06:22:23 on dist-test-slave-44d2"
server_key: "a1a567018d1e268f559ec38bde54b517"
server_key_iv: "d4067584a7da1019f52b42001d9431aa"
server_key_version: "encryptionkey@0"
I20260807 06:22:23.660377 17475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260807 06:22:23.663120 17493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:23.664602 17475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:22:23.664740 17475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "647a80a9fa0146c2b5abe8f23f2e803b"
format_stamp: "Formatted at 2026-08-07 06:22:23 on dist-test-slave-44d2"
server_key: "a1a567018d1e268f559ec38bde54b517"
server_key_iv: "d4067584a7da1019f52b42001d9431aa"
server_key_version: "encryptionkey@0"
I20260807 06:22:23.664858 17475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:23.678917 17475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:23.682322 17475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:23.682547 17475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:23.683168 17475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:23.684052 17475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:23.684130 17475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:23.684201 17475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:23.684239 17475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:23.694175 17475 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:40799
I20260807 06:22:23.694196 17659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:40799 every 8 connection(s)
I20260807 06:22:23.695204 17475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:22:23.702862 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 17475
I20260807 06:22:23.702955 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance
I20260807 06:22:23.703279 16376 external_mini_cluster.cc:1428] Setting key 8b8f4d2ba7340ca57fb4e9a1f47e9f3d
Aug 07 06:22:23 dist-test-slave-44d2 krb5kdc[17123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083743, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:23.710719 17255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:23.697331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:33609 (local address 127.15.254.62:43029)
0807 06:22:23.697538 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:23.697542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:23.698385 (+ 843us) server_negotiation.cc:407] Connection header received
0807 06:22:23.699860 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:23.699863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:23.699983 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:23.700068 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:23.701450 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:23.701997 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:23.703309 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:23.703615 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:23.703737 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:23.706575 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:23.706582 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:23.706585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:23.706618 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:23.708175 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:23.708962 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:23.708966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:23.708968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:23.709062 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:23.709441 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:23.709444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:23.709446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:23.709649 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:23.709765 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:23.710403 (+ 638us) server_negotiation.cc:299] Negotiation successful
0807 06:22:23.710536 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260807 06:22:23.711673 17662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:23.697732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:43029 (local address 127.15.254.2:33609)
0807 06:22:23.698218 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:23.698263 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:23.699618 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:23.700253 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:23.700260 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:23.700752 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:23.701296 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:23.701312 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:23.702148 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:23.702151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:23.703182 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:23.703193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:23.703805 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:23.703809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:23.704042 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:23.704810 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:23.704828 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:23.706368 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:23.708403 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:23.708411 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:23.708428 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:23.708845 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:23.709174 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:23.709177 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:23.709179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:23.709328 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:23.709765 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:23.709774 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:23.710093 (+ 319us) client_negotiation.cc:769] Sending connection context
0807 06:22:23.710337 (+ 244us) client_negotiation.cc:240] Negotiation successful
0807 06:22:23.710741 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":114,"threads_started":1}
I20260807 06:22:23.713012 17660 heartbeater.cc:347] Connected to a master server at 127.15.254.62:43029
I20260807 06:22:23.713306 17660 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:23.713837 17660 heartbeater.cc:511] Master 127.15.254.62:43029 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.254.3@KRBTEST.COM; defaulting to no policy
I20260807 06:22:23.715045 17196 ts_manager.cc:194] Registered new tserver with Master: 647a80a9fa0146c2b5abe8f23f2e803b (127.15.254.2:40799)
I20260807 06:22:23.715668 17196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.2@KRBTEST.COM'} at 127.15.254.2:33609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
I20260807 06:22:23.737576 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:0
--local_ip_for_outbound_sockets=127.15.254.3
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:43029
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:44331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:23.859815 17667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:23.860136 17667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:23.860188 17667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:23.864424 17667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:23.864573 17667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:23.864655 17667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:22:23.870226 17667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:44331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:43029
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:23.871332 17667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:23.872380 17667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:23.883074 17674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:23.883073 17673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:23.883216 17676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:23.883395 17667 server_base.cc:1034] running on GCE node
I20260807 06:22:23.883893 17667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:23.884670 17667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:23.885958 17667 hybrid_clock.cc:648] HybridClock initialized: now 1786083743885856 us; error 62 us; skew 500 ppm
Aug 07 06:22:23 dist-test-slave-44d2 krb5kdc[17123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083743, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:23.889281 17667 init.cc:377] Logged in from keytab as kudu/127.15.254.3@KRBTEST.COM (short username kudu)
I20260807 06:22:23.890182 17667 webserver.cc:460] Webserver started at http://127.15.254.3:44787/ using document root <none> and password file <none>
I20260807 06:22:23.890738 17667 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:23.890799 17667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:23.890980 17667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:23.893235 17667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-2/data/instance:
uuid: "9bc145b8d31c472fbced1950656158ad"
format_stamp: "Formatted at 2026-08-07 06:22:23 on dist-test-slave-44d2"
server_key: "716ecc238c4f73597513a7c7c1a4ed5f"
server_key_iv: "2a4657fc9f3f74f6dd5f0aad38483daa"
server_key_version: "encryptionkey@0"
I20260807 06:22:23.893788 17667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance:
uuid: "9bc145b8d31c472fbced1950656158ad"
format_stamp: "Formatted at 2026-08-07 06:22:23 on dist-test-slave-44d2"
server_key: "716ecc238c4f73597513a7c7c1a4ed5f"
server_key_iv: "2a4657fc9f3f74f6dd5f0aad38483daa"
server_key_version: "encryptionkey@0"
I20260807 06:22:23.898083 17667 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:22:23.900756 17686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:23.902047 17667 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 06:22:23.902195 17667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "9bc145b8d31c472fbced1950656158ad"
format_stamp: "Formatted at 2026-08-07 06:22:23 on dist-test-slave-44d2"
server_key: "716ecc238c4f73597513a7c7c1a4ed5f"
server_key_iv: "2a4657fc9f3f74f6dd5f0aad38483daa"
server_key_version: "encryptionkey@0"
I20260807 06:22:23.902314 17667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:23.927137 17667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:23.930724 17667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:23.930953 17667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:23.931663 17667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:23.932662 17667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:23.932742 17667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:23.932816 17667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:23.932874 17667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:23.943326 17667 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:33621
I20260807 06:22:23.943403 17857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:33621 every 8 connection(s)
I20260807 06:22:23.944555 17667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:22:23.945639 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 17667
I20260807 06:22:23.945741 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance
I20260807 06:22:23.945974 16376 external_mini_cluster.cc:1428] Setting key 5b44e609a66559735f398dedeb8ec775
Aug 07 06:22:23 dist-test-slave-44d2 krb5kdc[17123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083743, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:23.959898 17255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:23.946746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:34705 (local address 127.15.254.62:43029)
0807 06:22:23.947101 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:23.947108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:23.948121 (+ 1013us) server_negotiation.cc:407] Connection header received
0807 06:22:23.949661 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:23.949664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:23.949711 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:23.949785 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:23.951120 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:23.951664 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:23.953028 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:23.953248 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:23.953331 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:23.956129 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:23.956132 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:23.956135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:23.956160 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:23.957643 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:23.958165 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:23.958168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:23.958169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:23.958248 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:23.958570 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:23.958572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:23.958573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:23.958821 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:23.958926 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:23.959589 (+ 663us) server_negotiation.cc:299] Negotiation successful
0807 06:22:23.959727 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260807 06:22:23.961005 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:23.947243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:43029 (local address 127.15.254.3:34705)
0807 06:22:23.947933 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:23.947993 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:23.949453 (+ 1460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:23.949968 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:23.949976 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:23.950441 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:23.950980 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:23.950994 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:23.951806 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:23.951809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:23.952909 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:23.952920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:23.953390 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:23.953393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:23.953574 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:23.954198 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:23.954216 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:23.955919 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:23.957762 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:23.957768 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:23.957782 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:23.958070 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:23.958334 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:23.958336 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:23.958338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:23.958485 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:23.958951 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:23.958958 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:23.959291 (+ 333us) client_negotiation.cc:769] Sending connection context
0807 06:22:23.959515 (+ 224us) client_negotiation.cc:240] Negotiation successful
0807 06:22:23.959898 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":162,"threads_started":1}
I20260807 06:22:23.962306 17858 heartbeater.cc:347] Connected to a master server at 127.15.254.62:43029
I20260807 06:22:23.962587 17858 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:23.963124 17858 heartbeater.cc:511] Master 127.15.254.62:43029 requested a full tablet report, sending...
I20260807 06:22:23.964246 17196 ts_manager.cc:194] Registered new tserver with Master: 9bc145b8d31c472fbced1950656158ad (127.15.254.3:33621)
I20260807 06:22:23.964915 17196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.3@KRBTEST.COM'} at 127.15.254.3:34705
I20260807 06:22:23.972853 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:22:23.985456 17255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:23.977240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46928 (local address 127.15.254.62:43029)
0807 06:22:23.977408 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:23.977415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:23.977593 (+ 178us) server_negotiation.cc:407] Connection header received
0807 06:22:23.977792 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:23.977795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:23.977868 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:23.977947 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:23.978832 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:23.979532 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:23.980398 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:23.980722 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:23.980842 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:23.982310 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:23.982315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:23.982317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:23.982347 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:23.983614 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:23.984066 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:23.984071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:23.984073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:23.984165 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:23.984469 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:23.984473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:23.984475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:23.984745 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:23.984899 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:23.985155 (+ 256us) server_negotiation.cc:299] Negotiation successful
0807 06:22:23.985287 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 06:22:23.990521 17196 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:22:23.993791 17196 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:22:24.009752 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:24.002691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:33621 (local address 127.0.0.1:45836)
0807 06:22:24.003444 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:24.003500 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:24.003639 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:24.004991 (+ 1352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:24.004994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:24.005015 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:24.005121 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:24.005127 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:24.006551 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:24.006554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:24.007826 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:24.007837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:24.009096 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:24.009100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:24.009240 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:24.009294 (+ 54us) client_negotiation.cc:769] Sending connection context
0807 06:22:24.009430 (+ 136us) client_negotiation.cc:240] Negotiation successful
0807 06:22:24.009565 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":224,"spinlock_wait_cycles":12032,"thread_start_us":156,"threads_started":1}
I20260807 06:22:24.009819 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:24.002629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:35853 (local address 127.0.0.1:36686)
0807 06:22:24.003444 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:24.003500 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:24.003691 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:24.004431 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:24.004437 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:24.004458 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:24.004694 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:24.004708 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:24.006315 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:24.006318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:24.008072 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:24.008084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:24.009347 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:24.009350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:24.009480 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:24.009505 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 06:22:24.009580 (+ 75us) client_negotiation.cc:240] Negotiation successful
0807 06:22:24.009656 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":260,"thread_start_us":94,"threads_started":1}
I20260807 06:22:24.010426 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:24.003042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:40799 (local address 127.0.0.1:58042)
0807 06:22:24.004165 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:24.004181 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:24.004276 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:24.004810 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:24.004813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:24.004834 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:24.004970 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:24.004979 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:24.006653 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:24.006658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:24.008072 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:24.008084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:24.009681 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:24.009685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:24.009779 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:24.009807 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 06:22:24.010174 (+ 367us) client_negotiation.cc:240] Negotiation successful
0807 06:22:24.010253 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"mutex_wait_us":73,"thread_start_us":109,"threads_started":1}
I20260807 06:22:24.010432 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:24.002770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36686 (local address 127.15.254.1:35853)
0807 06:22:24.003201 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:24.003209 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:24.003622 (+ 413us) server_negotiation.cc:407] Connection header received
0807 06:22:24.003852 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:24.003858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:24.004091 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:24.004229 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:24.004845 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:24.006179 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:24.008219 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:24.009049 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:24.009159 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:24.009279 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:24.010159 (+ 880us) server_negotiation.cc:299] Negotiation successful
0807 06:22:24.010269 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":91,"threads_started":1}
I20260807 06:22:24.010859 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:24.003906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58042 (local address 127.15.254.2:40799)
0807 06:22:24.004271 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:24.004283 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:24.004309 (+ 26us) server_negotiation.cc:407] Connection header received
0807 06:22:24.004394 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:24.004400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:24.004625 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:24.004826 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:24.005098 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:24.006381 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:24.008597 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:24.009364 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:24.009464 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:24.009618 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:24.010609 (+ 991us) server_negotiation.cc:299] Negotiation successful
0807 06:22:24.010721 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":155,"threads_started":1}
I20260807 06:22:24.010902 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:24.003662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45836 (local address 127.15.254.3:33621)
0807 06:22:24.004428 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:24.004442 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:24.004461 (+ 19us) server_negotiation.cc:407] Connection header received
0807 06:22:24.004597 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:24.004601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:24.004776 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:24.005243 (+ 467us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:24.005371 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:24.006337 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:24.007987 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:24.008925 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:24.009050 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:24.009244 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:24.009482 (+ 238us) server_negotiation.cc:299] Negotiation successful
0807 06:22:24.010396 (+ 914us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":175,"threads_started":1}
I20260807 06:22:24.013737 17370 tablet_service.cc:1467] Processing CreateTablet for tablet 7d252bd14b11427dbbf9888b9cc4266e (DEFAULT_TABLE table=test-table [id=81d1aa8443e8419c9c56dfd7931e5789]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:24.013762 17568 tablet_service.cc:1467] Processing CreateTablet for tablet 7d252bd14b11427dbbf9888b9cc4266e (DEFAULT_TABLE table=test-table [id=81d1aa8443e8419c9c56dfd7931e5789]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:24.013698 17755 tablet_service.cc:1467] Processing CreateTablet for tablet 7d252bd14b11427dbbf9888b9cc4266e (DEFAULT_TABLE table=test-table [id=81d1aa8443e8419c9c56dfd7931e5789]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:24.015008 17755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d252bd14b11427dbbf9888b9cc4266e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:24.015017 17370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d252bd14b11427dbbf9888b9cc4266e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:24.015010 17568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d252bd14b11427dbbf9888b9cc4266e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:24.022593 17881 tablet_bootstrap.cc:492] T 7d252bd14b11427dbbf9888b9cc4266e P 9bc145b8d31c472fbced1950656158ad: Bootstrap starting.
I20260807 06:22:24.023978 17883 tablet_bootstrap.cc:492] T 7d252bd14b11427dbbf9888b9cc4266e P 647a80a9fa0146c2b5abe8f23f2e803b: Bootstrap starting.
I20260807 06:22:24.024957 17882 tablet_bootstrap.cc:492] T 7d252bd14b11427dbbf9888b9cc4266e P 45c9564409b14caabc2141c8a52ff4b3: Bootstrap starting.
I20260807 06:22:24.026278 17881 tablet_bootstrap.cc:654] T 7d252bd14b11427dbbf9888b9cc4266e P 9bc145b8d31c472fbced1950656158ad: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:24.026896 17883 tablet_bootstrap.cc:654] T 7d252bd14b11427dbbf9888b9cc4266e P 647a80a9fa0146c2b5abe8f23f2e803b: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:24.027410 17882 tablet_bootstrap.cc:654] T 7d252bd14b11427dbbf9888b9cc4266e P 45c9564409b14caabc2141c8a52ff4b3: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:24.027818 17883 log.cc:824] T 7d252bd14b11427dbbf9888b9cc4266e P 647a80a9fa0146c2b5abe8f23f2e803b: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:24.027818 17881 log.cc:824] T 7d252bd14b11427dbbf9888b9cc4266e P 9bc145b8d31c472fbced1950656158ad: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:24.028666 17882 log.cc:824] T 7d252bd14b11427dbbf9888b9cc4266e P 45c9564409b14caabc2141c8a52ff4b3: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:24.030113 17881 tablet_bootstrap.cc:492] T 7d252bd14b11427dbbf9888b9cc4266e P 9bc145b8d31c472fbced1950656158ad: No bootstrap required, opened a new log
I20260807 06:22:24.030342 17881 ts_tablet_manager.cc:1397] T 7d252bd14b11427dbbf9888b9cc4266e P 9bc145b8d31c472fbced1950656158ad: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260807 06:22:24.030344 17883 tablet_bootstrap.cc:492] T 7d252bd14b11427dbbf9888b9cc4266e P 647a80a9fa0146c2b5abe8f23f2e803b: No bootstrap required, opened a new log
I20260807 06:22:24.030565 17883 ts_tablet_manager.cc:1397] T 7d252bd14b11427dbbf9888b9cc4266e P 647a80a9fa0146c2b5abe8f23f2e803b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260807 06:22:24.030943 17882 tablet_bootstrap.cc:492] T 7d252bd14b11427dbbf9888b9cc4266e P 45c9564409b14caabc2141c8a52ff4b3: No bootstrap required, opened a new log
I20260807 06:22:24.031157 17882 ts_tablet_manager.cc:1397] T 7d252bd14b11427dbbf9888b9cc4266e P 45c9564409b14caabc2141c8a52ff4b3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260807 06:22:24.034251 17881 raft_consensus.cc:348] T 7d252bd14b11427dbbf9888b9cc4266e P 9bc145b8d31c472fbced1950656158ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c9564409b14caabc2141c8a52ff4b3" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35853 } } peers { permanent_uuid: "9bc145b8d31c472fbced1950656158ad" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33621 } } peers { permanent_uuid: "647a80a9fa0146c2b5abe8f23f2e803b" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40799 } }
I20260807 06:22:24.034531 17881 raft_consensus.cc:374] T 7d252bd14b11427dbbf9888b9cc4266e P 9bc145b8d31c472fbced1950656158ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:24.034590 17881 raft_consensus.cc:729] T 7d252bd14b11427dbbf9888b9cc4266e P 9bc145b8d31c472fbced1950656158ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc145b8d31c472fbced1950656158ad, State: Initialized, Role: FOLLOWER
I20260807 06:22:24.035043 17881 consensus_queue.cc:260] T 7d252bd14b11427dbbf9888b9cc4266e P 9bc145b8d31c472fbced1950656158ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c9564409b14caabc2141c8a52ff4b3" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35853 } } peers { permanent_uuid: "9bc145b8d31c472fbced1950656158ad" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33621 } } peers { permanent_uuid: "647a80a9fa0146c2b5abe8f23f2e803b" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40799 } }
I20260807 06:22:24.035776 17858 heartbeater.cc:503] Master 127.15.254.62:43029 was elected leader, sending a full tablet report...
I20260807 06:22:24.036175 17881 ts_tablet_manager.cc:1428] T 7d252bd14b11427dbbf9888b9cc4266e P 9bc145b8d31c472fbced1950656158ad: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260807 06:22:24.035950 17883 raft_consensus.cc:348] T 7d252bd14b11427dbbf9888b9cc4266e P 647a80a9fa0146c2b5abe8f23f2e803b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c9564409b14caabc2141c8a52ff4b3" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35853 } } peers { permanent_uuid: "9bc145b8d31c472fbced1950656158ad" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33621 } } peers { permanent_uuid: "647a80a9fa0146c2b5abe8f23f2e803b" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40799 } }
I20260807 06:22:24.036291 17883 raft_consensus.cc:374] T 7d252bd14b11427dbbf9888b9cc4266e P 647a80a9fa0146c2b5abe8f23f2e803b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:24.036357 17883 raft_consensus.cc:729] T 7d252bd14b11427dbbf9888b9cc4266e P 647a80a9fa0146c2b5abe8f23f2e803b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 647a80a9fa0146c2b5abe8f23f2e803b, State: Initialized, Role: FOLLOWER
I20260807 06:22:24.036901 17883 consensus_queue.cc:260] T 7d252bd14b11427dbbf9888b9cc4266e P 647a80a9fa0146c2b5abe8f23f2e803b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c9564409b14caabc2141c8a52ff4b3" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35853 } } peers { permanent_uuid: "9bc145b8d31c472fbced1950656158ad" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33621 } } peers { permanent_uuid: "647a80a9fa0146c2b5abe8f23f2e803b" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40799 } }
I20260807 06:22:24.036929 17882 raft_consensus.cc:348] T 7d252bd14b11427dbbf9888b9cc4266e P 45c9564409b14caabc2141c8a52ff4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c9564409b14caabc2141c8a52ff4b3" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35853 } } peers { permanent_uuid: "9bc145b8d31c472fbced1950656158ad" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33621 } } peers { permanent_uuid: "647a80a9fa0146c2b5abe8f23f2e803b" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40799 } }
I20260807 06:22:24.037223 17882 raft_consensus.cc:374] T 7d252bd14b11427dbbf9888b9cc4266e P 45c9564409b14caabc2141c8a52ff4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:24.037302 17882 raft_consensus.cc:729] T 7d252bd14b11427dbbf9888b9cc4266e P 45c9564409b14caabc2141c8a52ff4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45c9564409b14caabc2141c8a52ff4b3, State: Initialized, Role: FOLLOWER
I20260807 06:22:24.037797 17882 consensus_queue.cc:260] T 7d252bd14b11427dbbf9888b9cc4266e P 45c9564409b14caabc2141c8a52ff4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c9564409b14caabc2141c8a52ff4b3" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35853 } } peers { permanent_uuid: "9bc145b8d31c472fbced1950656158ad" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33621 } } peers { permanent_uuid: "647a80a9fa0146c2b5abe8f23f2e803b" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40799 } }
I20260807 06:22:24.037901 17660 heartbeater.cc:503] Master 127.15.254.62:43029 was elected leader, sending a full tablet report...
I20260807 06:22:24.038112 17883 ts_tablet_manager.cc:1428] T 7d252bd14b11427dbbf9888b9cc4266e P 647a80a9fa0146c2b5abe8f23f2e803b: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260807 06:22:24.038826 17882 ts_tablet_manager.cc:1428] T 7d252bd14b11427dbbf9888b9cc4266e P 45c9564409b14caabc2141c8a52ff4b3: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260807 06:22:24.039217 17469 heartbeater.cc:503] Master 127.15.254.62:43029 was elected leader, sending a full tablet report...
I20260807 06:22:24.045960 17889 raft_consensus.cc:482] T 7d252bd14b11427dbbf9888b9cc4266e P 45c9564409b14caabc2141c8a52ff4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:22:24.046128 17889 raft_consensus.cc:504] T 7d252bd14b11427dbbf9888b9cc4266e P 45c9564409b14caabc2141c8a52ff4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c9564409b14caabc2141c8a52ff4b3" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35853 } } peers { permanent_uuid: "9bc145b8d31c472fbced1950656158ad" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33621 } } peers { permanent_uuid: "647a80a9fa0146c2b5abe8f23f2e803b" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40799 } }
I20260807 06:22:24.047134 17889 leader_election.cc:288] T 7d252bd14b11427dbbf9888b9cc4266e P 45c9564409b14caabc2141c8a52ff4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc145b8d31c472fbced1950656158ad (127.15.254.3:33621), 647a80a9fa0146c2b5abe8f23f2e803b (127.15.254.2:40799)
I20260807 06:22:24.051323 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:24.047437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:58677 (local address 127.15.254.2:40799)
0807 06:22:24.047572 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:24.047577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:24.048015 (+ 438us) server_negotiation.cc:407] Connection header received
0807 06:22:24.048165 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:24.048168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:24.048215 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:24.048303 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:24.048694 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:24.049183 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:24.050357 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:24.050751 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:24.050887 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:24.050940 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:24.051135 (+ 195us) server_negotiation.cc:299] Negotiation successful
0807 06:22:24.051204 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260807 06:22:24.051421 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:24.047404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:42069 (local address 127.15.254.3:33621)
0807 06:22:24.047607 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:24.047612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:24.047955 (+ 343us) server_negotiation.cc:407] Connection header received
0807 06:22:24.048145 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:24.048149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:24.048199 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:24.048286 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:24.048612 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:24.049182 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:24.050367 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:24.050784 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:24.050944 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:24.050995 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:24.051235 (+ 240us) server_negotiation.cc:299] Negotiation successful
0807 06:22:24.051309 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260807 06:22:24.051360 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:24.047356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:40799 (local address 127.15.254.1:58677)
0807 06:22:24.047795 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:24.047806 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:24.048053 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:24.048325 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:24.048328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:24.048345 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:24.048579 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:24.048587 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:24.049301 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:24.049305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:24.050222 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:24.050232 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:24.050882 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:24.050886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:24.050991 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:24.051021 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 06:22:24.051117 (+ 96us) client_negotiation.cc:240] Negotiation successful
0807 06:22:24.051187 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":33,"thread_start_us":160,"threads_started":1}
I20260807 06:22:24.052095 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:24.047351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:33621 (local address 127.15.254.1:42069)
0807 06:22:24.047741 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:24.047754 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:24.048021 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:24.048298 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:24.048301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:24.048318 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:24.048469 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:24.048475 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:24.049306 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:24.049310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:24.050232 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:24.050242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:24.050938 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:24.050941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:24.051095 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:24.051114 (+ 19us) client_negotiation.cc:769] Sending connection context
0807 06:22:24.051204 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 06:22:24.051275 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":70,"thread_start_us":154,"threads_started":1}
I20260807 06:22:24.052520 17595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d252bd14b11427dbbf9888b9cc4266e" candidate_uuid: "45c9564409b14caabc2141c8a52ff4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "647a80a9fa0146c2b5abe8f23f2e803b" is_pre_election: true
I20260807 06:22:24.052889 17595 raft_consensus.cc:2454] T 7d252bd14b11427dbbf9888b9cc4266e P 647a80a9fa0146c2b5abe8f23f2e803b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45c9564409b14caabc2141c8a52ff4b3 in term 0.
I20260807 06:22:24.052892 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d252bd14b11427dbbf9888b9cc4266e" candidate_uuid: "45c9564409b14caabc2141c8a52ff4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc145b8d31c472fbced1950656158ad" is_pre_election: true
I20260807 06:22:24.053193 17785 raft_consensus.cc:2454] T 7d252bd14b11427dbbf9888b9cc4266e P 9bc145b8d31c472fbced1950656158ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45c9564409b14caabc2141c8a52ff4b3 in term 0.
I20260807 06:22:24.053493 17308 leader_election.cc:302] T 7d252bd14b11427dbbf9888b9cc4266e P 45c9564409b14caabc2141c8a52ff4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45c9564409b14caabc2141c8a52ff4b3, 647a80a9fa0146c2b5abe8f23f2e803b; no voters:
I20260807 06:22:24.053825 17889 raft_consensus.cc:2781] T 7d252bd14b11427dbbf9888b9cc4266e P 45c9564409b14caabc2141c8a52ff4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:22:24.053911 17889 raft_consensus.cc:482] T 7d252bd14b11427dbbf9888b9cc4266e P 45c9564409b14caabc2141c8a52ff4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:22:24.053944 17889 raft_consensus.cc:3037] T 7d252bd14b11427dbbf9888b9cc4266e P 45c9564409b14caabc2141c8a52ff4b3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:24.055176 17889 raft_consensus.cc:504] T 7d252bd14b11427dbbf9888b9cc4266e P 45c9564409b14caabc2141c8a52ff4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c9564409b14caabc2141c8a52ff4b3" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35853 } } peers { permanent_uuid: "9bc145b8d31c472fbced1950656158ad" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33621 } } peers { permanent_uuid: "647a80a9fa0146c2b5abe8f23f2e803b" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40799 } }
I20260807 06:22:24.055630 17889 leader_election.cc:288] T 7d252bd14b11427dbbf9888b9cc4266e P 45c9564409b14caabc2141c8a52ff4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc145b8d31c472fbced1950656158ad (127.15.254.3:33621), 647a80a9fa0146c2b5abe8f23f2e803b (127.15.254.2:40799)
I20260807 06:22:24.056089 17595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d252bd14b11427dbbf9888b9cc4266e" candidate_uuid: "45c9564409b14caabc2141c8a52ff4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "647a80a9fa0146c2b5abe8f23f2e803b"
I20260807 06:22:24.056089 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d252bd14b11427dbbf9888b9cc4266e" candidate_uuid: "45c9564409b14caabc2141c8a52ff4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc145b8d31c472fbced1950656158ad"
I20260807 06:22:24.056227 17785 raft_consensus.cc:3037] T 7d252bd14b11427dbbf9888b9cc4266e P 9bc145b8d31c472fbced1950656158ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:24.056226 17595 raft_consensus.cc:3037] T 7d252bd14b11427dbbf9888b9cc4266e P 647a80a9fa0146c2b5abe8f23f2e803b [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:24.057788 17595 raft_consensus.cc:2454] T 7d252bd14b11427dbbf9888b9cc4266e P 647a80a9fa0146c2b5abe8f23f2e803b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45c9564409b14caabc2141c8a52ff4b3 in term 1.
I20260807 06:22:24.057788 17785 raft_consensus.cc:2454] T 7d252bd14b11427dbbf9888b9cc4266e P 9bc145b8d31c472fbced1950656158ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45c9564409b14caabc2141c8a52ff4b3 in term 1.
I20260807 06:22:24.058192 17307 leader_election.cc:302] T 7d252bd14b11427dbbf9888b9cc4266e P 45c9564409b14caabc2141c8a52ff4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45c9564409b14caabc2141c8a52ff4b3, 9bc145b8d31c472fbced1950656158ad; no voters:
I20260807 06:22:24.058421 17889 raft_consensus.cc:2781] T 7d252bd14b11427dbbf9888b9cc4266e P 45c9564409b14caabc2141c8a52ff4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:24.058686 17889 raft_consensus.cc:686] T 7d252bd14b11427dbbf9888b9cc4266e P 45c9564409b14caabc2141c8a52ff4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 45c9564409b14caabc2141c8a52ff4b3, State: Running, Role: LEADER
I20260807 06:22:24.059039 17889 consensus_queue.cc:237] T 7d252bd14b11427dbbf9888b9cc4266e P 45c9564409b14caabc2141c8a52ff4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c9564409b14caabc2141c8a52ff4b3" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35853 } } peers { permanent_uuid: "9bc145b8d31c472fbced1950656158ad" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33621 } } peers { permanent_uuid: "647a80a9fa0146c2b5abe8f23f2e803b" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40799 } }
I20260807 06:22:24.062634 17192 catalog_manager.cc:5495] T 7d252bd14b11427dbbf9888b9cc4266e P 45c9564409b14caabc2141c8a52ff4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45c9564409b14caabc2141c8a52ff4b3 (127.15.254.1). New cstate: current_term: 1 leader_uuid: "45c9564409b14caabc2141c8a52ff4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c9564409b14caabc2141c8a52ff4b3" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bc145b8d31c472fbced1950656158ad" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "647a80a9fa0146c2b5abe8f23f2e803b" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40799 } health_report { overall_health: UNKNOWN } } }
Aug 07 06:22:24 dist-test-slave-44d2 krb5kdc[17123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.1@KRBTEST.COM
I20260807 06:22:24.095990 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:24.082214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36690 (local address 127.15.254.1:35853)
0807 06:22:24.082404 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:24.082410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:24.082568 (+ 158us) server_negotiation.cc:407] Connection header received
0807 06:22:24.082735 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:24.082738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:24.082789 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:24.082887 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:24.083608 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:24.084126 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:24.085129 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:24.085379 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:24.085508 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:24.092347 (+ 6839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:24.092361 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:24.092379 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:24.092406 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:24.094144 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:24.094606 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:24.094611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:24.094613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:24.094697 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:24.094960 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:24.094962 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:24.094963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:24.095292 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:24.095501 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:24.095626 (+ 125us) server_negotiation.cc:299] Negotiation successful
0807 06:22:24.095738 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
Aug 07 06:22:24 dist-test-slave-44d2 krb5kdc[17123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083744, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 06:22:24 dist-test-slave-44d2 krb5kdc[17123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083744, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.254.1@KRBTEST.COM
I20260807 06:22:24.126561 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:24.117374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36702 (local address 127.15.254.1:35853)
0807 06:22:24.117551 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:24.117557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:24.117735 (+ 178us) server_negotiation.cc:407] Connection header received
0807 06:22:24.117928 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:24.117932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:24.117987 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:24.118106 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:24.118864 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:24.119664 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:24.120568 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:24.120798 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:24.120885 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:24.123417 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:24.123420 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:24.123422 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:24.123446 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:24.124904 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:24.125352 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:24.125355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:24.125356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:24.125434 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:24.125692 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:24.125694 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:24.125695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:24.125857 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:24.125980 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:24.126228 (+ 248us) server_negotiation.cc:299] Negotiation successful
0807 06:22:24.126332 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Aug 07 06:22:24 dist-test-slave-44d2 krb5kdc[17123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 06:22:24 dist-test-slave-44d2 krb5kdc[17123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.1@KRBTEST.COM
I20260807 06:22:24.157289 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:24.147927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36704 (local address 127.15.254.1:35853)
0807 06:22:24.148151 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:24.148155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:24.148369 (+ 214us) server_negotiation.cc:407] Connection header received
0807 06:22:24.148616 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:24.148619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:24.148672 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:24.148749 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:24.149749 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:24.150259 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:24.151123 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:24.151333 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:24.151411 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:24.154116 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:24.154120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:24.154123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:24.154150 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:24.155696 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:24.156102 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:24.156105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:24.156107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:24.156195 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:24.156444 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:24.156447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:24.156449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:24.156673 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:24.156824 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:24.157011 (+ 187us) server_negotiation.cc:299] Negotiation successful
0807 06:22:24.157133 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
W20260807 06:22:24.158991 17340 tablet_service.cc:2908] Rejecting scan request for tablet 7d252bd14b11427dbbf9888b9cc4266e: Uninitialized: clean time has not yet been initialized
I20260807 06:22:24.161221 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 17286
I20260807 06:22:24.176810 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 17475
I20260807 06:22:24.189080 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 17667
I20260807 06:22:24.203871 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 17145
2026-08-07T06:22:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2635 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:22:24 dist-test-slave-44d2 krb5kdc[17916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:22:24 dist-test-slave-44d2 krb5kdc[17916](info): set up 2 sockets
Aug 07 06:22:24 dist-test-slave-44d2 krb5kdc[17916](info): commencing operation
krb5kdc: starting...
W20260807 06:22:25.439308 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:22:25 dist-test-slave-44d2 krb5kdc[17916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:22:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:22:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:22:25.576443 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:39757
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:39757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:25.696518 17933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:25.696897 17933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:25.696978 17933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:25.701144 17933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:22:25.701261 17933 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 06:22:25.701294 17933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:25.701341 17933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:25.701377 17933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:22:25.701414 17933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:22:25.706887 17933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:39757
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:39757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:25.708148 17933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:25.709322 17933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:25.718791 17946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:25.718791 17943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:25.718835 17944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:25.719144 17933 server_base.cc:1034] running on GCE node
I20260807 06:22:25.719787 17933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:25.721057 17933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:25.722287 17933 hybrid_clock.cc:648] HybridClock initialized: now 1786083745722267 us; error 32 us; skew 500 ppm
Aug 07 06:22:25 dist-test-slave-44d2 krb5kdc[17916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083745, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:25.725525 17933 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:22:25.726507 17933 webserver.cc:460] Webserver started at http://127.15.254.62:44431/ using document root <none> and password file <none>
I20260807 06:22:25.727072 17933 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:25.727135 17933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:25.727315 17933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:25.729494 17933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "099cd300fbd84bcfbcd9554503e7a55f"
format_stamp: "Formatted at 2026-08-07 06:22:25 on dist-test-slave-44d2"
server_key: "0016aa4cc0f88efb529a4bde63efcf3f"
server_key_iv: "0624874e3d74516abd9f4b3188425610"
server_key_version: "encryptionkey@0"
I20260807 06:22:25.730043 17933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "099cd300fbd84bcfbcd9554503e7a55f"
format_stamp: "Formatted at 2026-08-07 06:22:25 on dist-test-slave-44d2"
server_key: "0016aa4cc0f88efb529a4bde63efcf3f"
server_key_iv: "0624874e3d74516abd9f4b3188425610"
server_key_version: "encryptionkey@0"
I20260807 06:22:25.734010 17933 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:22:25.736742 17955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:25.738157 17933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:22:25.738273 17933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "099cd300fbd84bcfbcd9554503e7a55f"
format_stamp: "Formatted at 2026-08-07 06:22:25 on dist-test-slave-44d2"
server_key: "0016aa4cc0f88efb529a4bde63efcf3f"
server_key_iv: "0624874e3d74516abd9f4b3188425610"
server_key_version: "encryptionkey@0"
I20260807 06:22:25.738361 17933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:25.764179 17933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:25.767819 17933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:25.768042 17933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:25.776989 17933 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:39757
I20260807 06:22:25.777012 18043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:39757 every 8 connection(s)
I20260807 06:22:25.778057 17933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:22:25.781111 18044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:25.784776 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 17933
I20260807 06:22:25.784875 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:22:25.785136 16376 external_mini_cluster.cc:1428] Setting key 2a3c8066ead2a4d178b061f449c5e515
I20260807 06:22:25.787185 18044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 099cd300fbd84bcfbcd9554503e7a55f: Bootstrap starting.
I20260807 06:22:25.789969 18044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 099cd300fbd84bcfbcd9554503e7a55f: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:25.790887 18044 log.cc:824] T 00000000000000000000000000000000 P 099cd300fbd84bcfbcd9554503e7a55f: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:25.793787 18044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 099cd300fbd84bcfbcd9554503e7a55f: No bootstrap required, opened a new log
Aug 07 06:22:25 dist-test-slave-44d2 krb5kdc[17916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:25.797669 18044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 099cd300fbd84bcfbcd9554503e7a55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "099cd300fbd84bcfbcd9554503e7a55f" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39757 } }
I20260807 06:22:25.797904 18044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 099cd300fbd84bcfbcd9554503e7a55f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:25.797966 18044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 099cd300fbd84bcfbcd9554503e7a55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 099cd300fbd84bcfbcd9554503e7a55f, State: Initialized, Role: FOLLOWER
I20260807 06:22:25.798466 18044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 099cd300fbd84bcfbcd9554503e7a55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "099cd300fbd84bcfbcd9554503e7a55f" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39757 } }
I20260807 06:22:25.798596 18044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 099cd300fbd84bcfbcd9554503e7a55f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:22:25.798669 18044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 099cd300fbd84bcfbcd9554503e7a55f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:22:25.798792 18044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 099cd300fbd84bcfbcd9554503e7a55f [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:25.799727 18044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 099cd300fbd84bcfbcd9554503e7a55f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "099cd300fbd84bcfbcd9554503e7a55f" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39757 } }
I20260807 06:22:25.800099 18044 leader_election.cc:302] T 00000000000000000000000000000000 P 099cd300fbd84bcfbcd9554503e7a55f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 099cd300fbd84bcfbcd9554503e7a55f; no voters:
I20260807 06:22:25.800468 18044 leader_election.cc:288] T 00000000000000000000000000000000 P 099cd300fbd84bcfbcd9554503e7a55f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:22:25.800609 18050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 099cd300fbd84bcfbcd9554503e7a55f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:25.800818 18050 raft_consensus.cc:686] T 00000000000000000000000000000000 P 099cd300fbd84bcfbcd9554503e7a55f [term 1 LEADER]: Becoming Leader. State: Replica: 099cd300fbd84bcfbcd9554503e7a55f, State: Running, Role: LEADER
I20260807 06:22:25.801123 18050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 099cd300fbd84bcfbcd9554503e7a55f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "099cd300fbd84bcfbcd9554503e7a55f" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39757 } }
I20260807 06:22:25.801865 18044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 099cd300fbd84bcfbcd9554503e7a55f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:22:25.802408 18050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 099cd300fbd84bcfbcd9554503e7a55f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 099cd300fbd84bcfbcd9554503e7a55f. Latest consensus state: current_term: 1 leader_uuid: "099cd300fbd84bcfbcd9554503e7a55f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "099cd300fbd84bcfbcd9554503e7a55f" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39757 } } }
I20260807 06:22:25.802513 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:25.786688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42954 (local address 127.15.254.62:39757)
0807 06:22:25.787137 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:25.787158 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:25.787202 (+ 44us) server_negotiation.cc:407] Connection header received
0807 06:22:25.787997 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:25.788020 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:25.788471 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:25.788871 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:25.789565 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:25.791108 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:25.792334 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:25.793170 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:25.793345 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:25.796425 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:25.796437 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:25.796452 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:25.796539 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:25.798674 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:25.799247 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:25.799253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:25.799260 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:25.799393 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:25.799716 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:25.799720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:25.799722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:25.800250 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:25.800452 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:25.801269 (+ 817us) server_negotiation.cc:299] Negotiation successful
0807 06:22:25.801562 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":130,"threads_started":1}
I20260807 06:22:25.802520 18050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 099cd300fbd84bcfbcd9554503e7a55f [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:25.802899 18051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 099cd300fbd84bcfbcd9554503e7a55f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "099cd300fbd84bcfbcd9554503e7a55f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "099cd300fbd84bcfbcd9554503e7a55f" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39757 } } }
I20260807 06:22:25.802999 18051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 099cd300fbd84bcfbcd9554503e7a55f [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:25.803036 18057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:22:25.806468 18057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:22:25.812170 18057 catalog_manager.cc:1347] Generated new cluster ID: 02079a031ce04ac79e17f6aef3e8d01b
I20260807 06:22:25.812251 18057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:22:25.826140 18057 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:22:25.827227 18057 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:22:25.832337 18057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 099cd300fbd84bcfbcd9554503e7a55f: Generated new TSK 0
I20260807 06:22:25.833081 18057 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
I20260807 06:22:25.868691 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:0
--local_ip_for_outbound_sockets=127.15.254.1
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:39757
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:25.990034 18081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:25.990309 18081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:25.990358 18081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:25.994462 18081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:25.994529 18081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:25.994571 18081 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 06:22:25.994659 18081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:22:25.999894 18081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.254.62:39757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:26.001118 18081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:26.002287 18081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:26.012825 18086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:26.012887 18087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:26.012835 18093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:26.013295 18081 server_base.cc:1034] running on GCE node
I20260807 06:22:26.014097 18081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:26.014904 18081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:26.016086 18081 hybrid_clock.cc:648] HybridClock initialized: now 1786083746016066 us; error 32 us; skew 500 ppm
Aug 07 06:22:26 dist-test-slave-44d2 krb5kdc[17916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083746, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:26.019661 18081 init.cc:377] Logged in from keytab as kudu/127.15.254.1@KRBTEST.COM (short username kudu)
I20260807 06:22:26.020819 18081 webserver.cc:460] Webserver started at http://127.15.254.1:44133/ using document root <none> and password file <none>
I20260807 06:22:26.021447 18081 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:26.021515 18081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:26.021776 18081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:26.023932 18081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-0/data/instance:
uuid: "f0654852d6404e6488ebce8e50bf9943"
format_stamp: "Formatted at 2026-08-07 06:22:26 on dist-test-slave-44d2"
server_key: "a0eb70e5daf92e11f1648475eada7c90"
server_key_iv: "26d2bbabf610addb379a3e15503f7a9f"
server_key_version: "encryptionkey@0"
I20260807 06:22:26.024601 18081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance:
uuid: "f0654852d6404e6488ebce8e50bf9943"
format_stamp: "Formatted at 2026-08-07 06:22:26 on dist-test-slave-44d2"
server_key: "a0eb70e5daf92e11f1648475eada7c90"
server_key_iv: "26d2bbabf610addb379a3e15503f7a9f"
server_key_version: "encryptionkey@0"
I20260807 06:22:26.028810 18081 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:22:26.031525 18108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:26.033020 18081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:22:26.033192 18081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "f0654852d6404e6488ebce8e50bf9943"
format_stamp: "Formatted at 2026-08-07 06:22:26 on dist-test-slave-44d2"
server_key: "a0eb70e5daf92e11f1648475eada7c90"
server_key_iv: "26d2bbabf610addb379a3e15503f7a9f"
server_key_version: "encryptionkey@0"
I20260807 06:22:26.033299 18081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:26.063336 18081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:26.066823 18081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:26.067065 18081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:26.068693 18081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:26.068746 18081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:26.068790 18081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:26.068849 18081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 06:22:26 dist-test-slave-44d2 krb5kdc[17916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083746, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:26.079982 18081 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:36583
I20260807 06:22:26.080098 18264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:36583 every 8 connection(s)
I20260807 06:22:26.080965 18081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:22:26.084100 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:26.069767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:33839 (local address 127.15.254.62:39757)
0807 06:22:26.069943 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:26.069948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:26.070825 (+ 877us) server_negotiation.cc:407] Connection header received
0807 06:22:26.072106 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:26.072109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:26.072178 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:26.072299 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:26.074137 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.075002 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.076969 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.077235 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.077321 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:26.080591 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:26.080595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:26.080597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:26.080626 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:26.082250 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:26.082745 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:26.082748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:26.082749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:26.082830 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:26.083143 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:26.083146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:26.083147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:26.083321 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:26.083395 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:26.083862 (+ 467us) server_negotiation.cc:299] Negotiation successful
0807 06:22:26.083966 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 06:22:26.084951 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:26.070138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39757 (local address 127.15.254.1:33839)
0807 06:22:26.070642 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:26.070695 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:26.071745 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:26.072601 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:26.072615 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:26.073147 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:26.073910 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:26.073934 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.075160 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.075166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:26.076797 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:26.076809 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.077412 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.077419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:26.077656 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:26.078411 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:26.078436 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:26.080384 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:26.082355 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:26.082359 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:26.082370 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:26.082636 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:26.082912 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:26.082914 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:26.082915 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:26.083048 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:26.083412 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:26.083417 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:26.083649 (+ 232us) client_negotiation.cc:769] Sending connection context
0807 06:22:26.083796 (+ 147us) client_negotiation.cc:240] Negotiation successful
0807 06:22:26.084035 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":142,"threads_started":1}
I20260807 06:22:26.086273 18265 heartbeater.cc:347] Connected to a master server at 127.15.254.62:39757
I20260807 06:22:26.086452 18265 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:26.086959 18265 heartbeater.cc:511] Master 127.15.254.62:39757 requested a full tablet report, sending...
I20260807 06:22:26.087415 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 18081
I20260807 06:22:26.087538 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance
I20260807 06:22:26.087850 16376 external_mini_cluster.cc:1428] Setting key 8ac15acff0d3043bdb4eae5fc0f056ba
I20260807 06:22:26.088775 17982 ts_manager.cc:194] Registered new tserver with Master: f0654852d6404e6488ebce8e50bf9943 (127.15.254.1:36583)
I20260807 06:22:26.090137 17982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.1@KRBTEST.COM'} at 127.15.254.1:33839
I20260807 06:22:26.097110 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:26.088996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39757 (local address 127.15.254.1:53555)
0807 06:22:26.089319 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:26.089335 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:26.089436 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:26.089691 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:26.089695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:26.089928 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:26.090085 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:26.090096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.091066 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.091072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:26.091920 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:26.091931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.092432 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.092436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:26.092587 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:26.093124 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:26.093136 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:26.093460 (+ 324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:26.095385 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:26.095388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:26.095391 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:26.095617 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:26.095936 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:26.095939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:26.095941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:26.096036 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:26.096472 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:26.096475 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:26.096615 (+ 140us) client_negotiation.cc:769] Sending connection context
0807 06:22:26.096745 (+ 130us) client_negotiation.cc:240] Negotiation successful
0807 06:22:26.096910 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":125,"threads_started":1}
I20260807 06:22:26.097105 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:26.088801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:53555 (local address 127.15.254.62:39757)
0807 06:22:26.088908 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:26.088912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:26.089422 (+ 510us) server_negotiation.cc:407] Connection header received
0807 06:22:26.089514 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:26.089517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:26.089560 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:26.089642 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:26.090223 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.090915 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.092055 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.092297 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.092412 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:26.093658 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:26.093662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:26.093664 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:26.093696 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:26.095249 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:26.095738 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:26.095741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:26.095743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:26.095831 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:26.096151 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:26.096156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:26.096157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:26.096363 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:26.096482 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:26.096802 (+ 320us) server_negotiation.cc:299] Negotiation successful
0807 06:22:26.096925 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
WARNING: no policy specified for kudu/127.15.254.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
I20260807 06:22:26.120748 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:0
--local_ip_for_outbound_sockets=127.15.254.2
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:39757
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:26.236997 18278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:26.237298 18278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:26.237385 18278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:26.241425 18278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:26.241511 18278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:26.241595 18278 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 06:22:26.241685 18278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:22:26.246995 18278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.254.62:39757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:26.248136 18278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:26.249256 18278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 06:22:26.259917 18278 server_base.cc:1034] running on GCE node
W20260807 06:22:26.260017 18288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:26.259877 18290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:26.259877 18286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:26.260742 18278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:26.261433 18278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:26.262599 18278 hybrid_clock.cc:648] HybridClock initialized: now 1786083746262563 us; error 48 us; skew 500 ppm
Aug 07 06:22:26 dist-test-slave-44d2 krb5kdc[17916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083746, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:26.265630 18278 init.cc:377] Logged in from keytab as kudu/127.15.254.2@KRBTEST.COM (short username kudu)
I20260807 06:22:26.266556 18278 webserver.cc:460] Webserver started at http://127.15.254.2:44937/ using document root <none> and password file <none>
I20260807 06:22:26.267179 18278 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:26.267246 18278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:26.267488 18278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:26.269671 18278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-1/data/instance:
uuid: "3a95e762028341a59c1f85991a16607a"
format_stamp: "Formatted at 2026-08-07 06:22:26 on dist-test-slave-44d2"
server_key: "42bd984afdb37aabf0d908fd8d2b3fb3"
server_key_iv: "913ce08279da9a158769e711684541c5"
server_key_version: "encryptionkey@0"
I20260807 06:22:26.270254 18278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance:
uuid: "3a95e762028341a59c1f85991a16607a"
format_stamp: "Formatted at 2026-08-07 06:22:26 on dist-test-slave-44d2"
server_key: "42bd984afdb37aabf0d908fd8d2b3fb3"
server_key_iv: "913ce08279da9a158769e711684541c5"
server_key_version: "encryptionkey@0"
I20260807 06:22:26.274210 18278 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:22:26.276774 18299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:26.278087 18278 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 06:22:26.278240 18278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "3a95e762028341a59c1f85991a16607a"
format_stamp: "Formatted at 2026-08-07 06:22:26 on dist-test-slave-44d2"
server_key: "42bd984afdb37aabf0d908fd8d2b3fb3"
server_key_iv: "913ce08279da9a158769e711684541c5"
server_key_version: "encryptionkey@0"
I20260807 06:22:26.278362 18278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:26.293325 18278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:26.296427 18278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:26.296715 18278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:26.298329 18278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:26.298409 18278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:26.298486 18278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:26.298523 18278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 06:22:26 dist-test-slave-44d2 krb5kdc[17916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083746, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:26.310441 18278 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:40227
I20260807 06:22:26.310575 18468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:40227 every 8 connection(s)
I20260807 06:22:26.311614 18278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:22:26.313647 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:26.299180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:41243 (local address 127.15.254.62:39757)
0807 06:22:26.299350 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:26.299356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:26.300351 (+ 995us) server_negotiation.cc:407] Connection header received
0807 06:22:26.301465 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:26.301470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:26.301538 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:26.301635 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:26.303746 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.304470 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.305846 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.306185 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.306303 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:26.309359 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:26.309363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:26.309366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:26.309393 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:26.311132 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:26.311757 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:26.311761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:26.311769 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:26.311856 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:26.312315 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:26.312319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:26.312321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:26.312639 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:26.312767 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:26.313315 (+ 548us) server_negotiation.cc:299] Negotiation successful
0807 06:22:26.313456 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 06:22:26.314443 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:26.299666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39757 (local address 127.15.254.2:41243)
0807 06:22:26.300168 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:26.300221 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:26.301206 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:26.302192 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:26.302201 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:26.302666 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:26.303492 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:26.303516 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.304678 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.304682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:26.305712 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:26.305722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.306392 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.306399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:26.306562 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:26.307242 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:26.307259 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:26.309163 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:26.311262 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:26.311267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:26.311307 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:26.311626 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:26.311963 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:26.311966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:26.311968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:26.312191 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:26.312766 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:26.312775 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:26.313031 (+ 256us) client_negotiation.cc:769] Sending connection context
0807 06:22:26.313265 (+ 234us) client_negotiation.cc:240] Negotiation successful
0807 06:22:26.313605 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":135,"threads_started":1}
I20260807 06:22:26.315868 18469 heartbeater.cc:347] Connected to a master server at 127.15.254.62:39757
I20260807 06:22:26.316044 18469 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:26.316653 18469 heartbeater.cc:511] Master 127.15.254.62:39757 requested a full tablet report, sending...
I20260807 06:22:26.317847 17982 ts_manager.cc:194] Registered new tserver with Master: 3a95e762028341a59c1f85991a16607a (127.15.254.2:40227)
I20260807 06:22:26.318080 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 18278
I20260807 06:22:26.318198 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance
I20260807 06:22:26.318516 16376 external_mini_cluster.cc:1428] Setting key 6897b260d7995081daf322d7a7011599
I20260807 06:22:26.318702 17982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.2@KRBTEST.COM'} at 127.15.254.2:41243
I20260807 06:22:26.326762 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:26.318289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:33289 (local address 127.15.254.62:39757)
0807 06:22:26.318411 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:26.318415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:26.318724 (+ 309us) server_negotiation.cc:407] Connection header received
0807 06:22:26.318898 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:26.318901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:26.318962 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:26.319049 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:26.319627 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.320139 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.321107 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.321331 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.321452 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:26.322886 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:26.322889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:26.322891 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:26.322915 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:26.324777 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:26.325215 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:26.325221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:26.325226 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:26.325334 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:26.325624 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:26.325630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:26.325635 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:26.325983 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:26.326110 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:26.326446 (+ 336us) server_negotiation.cc:299] Negotiation successful
0807 06:22:26.326589 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 06:22:26.326779 18475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:26.318324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39757 (local address 127.15.254.2:33289)
0807 06:22:26.318617 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:26.318633 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:26.318743 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:26.319168 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:26.319171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:26.319392 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:26.319503 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:26.319509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.320278 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.320288 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:26.320933 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:26.320944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.321455 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.321459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:26.321556 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:26.322205 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:26.322227 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:26.322733 (+ 506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:26.324896 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:26.324898 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:26.324900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:26.325108 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:26.325441 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:26.325443 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:26.325445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:26.325521 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:26.326162 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:26.326167 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:26.326279 (+ 112us) client_negotiation.cc:769] Sending connection context
0807 06:22:26.326366 (+ 87us) client_negotiation.cc:240] Negotiation successful
0807 06:22:26.326480 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for kudu/127.15.254.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
I20260807 06:22:26.351934 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:0
--local_ip_for_outbound_sockets=127.15.254.3
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:39757
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:26.469311 18482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:26.469581 18482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:26.469627 18482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:26.473856 18482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:26.473928 18482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:26.474007 18482 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 06:22:26.474076 18482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:22:26.479461 18482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.254.62:39757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:26.480660 18482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:26.481801 18482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:26.492100 18489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:26.492205 18490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:26.492291 18482 server_base.cc:1034] running on GCE node
W20260807 06:22:26.492100 18492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:26.492786 18482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:26.493450 18482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:26.494637 18482 hybrid_clock.cc:648] HybridClock initialized: now 1786083746494604 us; error 45 us; skew 500 ppm
Aug 07 06:22:26 dist-test-slave-44d2 krb5kdc[17916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083746, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:26.497830 18482 init.cc:377] Logged in from keytab as kudu/127.15.254.3@KRBTEST.COM (short username kudu)
I20260807 06:22:26.498826 18482 webserver.cc:460] Webserver started at http://127.15.254.3:41771/ using document root <none> and password file <none>
I20260807 06:22:26.499429 18482 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:26.499526 18482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:26.499778 18482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:26.502110 18482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-2/data/instance:
uuid: "588e6c5989f04539b084ee778e9ed8a7"
format_stamp: "Formatted at 2026-08-07 06:22:26 on dist-test-slave-44d2"
server_key: "1e6fe8af286fb51394f3613e81329adb"
server_key_iv: "8efa50dbc467bf8a139c545f4aa91f3b"
server_key_version: "encryptionkey@0"
I20260807 06:22:26.502786 18482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance:
uuid: "588e6c5989f04539b084ee778e9ed8a7"
format_stamp: "Formatted at 2026-08-07 06:22:26 on dist-test-slave-44d2"
server_key: "1e6fe8af286fb51394f3613e81329adb"
server_key_iv: "8efa50dbc467bf8a139c545f4aa91f3b"
server_key_version: "encryptionkey@0"
I20260807 06:22:26.506784 18482 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:22:26.509430 18502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:26.510677 18482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 06:22:26.510821 18482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "588e6c5989f04539b084ee778e9ed8a7"
format_stamp: "Formatted at 2026-08-07 06:22:26 on dist-test-slave-44d2"
server_key: "1e6fe8af286fb51394f3613e81329adb"
server_key_iv: "8efa50dbc467bf8a139c545f4aa91f3b"
server_key_version: "encryptionkey@0"
I20260807 06:22:26.510937 18482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:26.525585 18482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:26.529829 18482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:26.530097 18482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:26.531813 18482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:26.531914 18482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:26.531996 18482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:26.532058 18482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 06:22:26 dist-test-slave-44d2 krb5kdc[17916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083746, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:26.545347 18482 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:45095
I20260807 06:22:26.545492 18676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:45095 every 8 connection(s)
I20260807 06:22:26.546689 18482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:22:26.549477 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:26.532959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:36031 (local address 127.15.254.62:39757)
0807 06:22:26.533140 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:26.533145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:26.534242 (+ 1097us) server_negotiation.cc:407] Connection header received
0807 06:22:26.535818 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:26.535825 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:26.535902 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:26.536019 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:26.538011 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.538707 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.540022 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.540262 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.540391 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:26.545178 (+ 4787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:26.545181 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:26.545183 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:26.545210 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:26.547014 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:26.547671 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:26.547676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:26.547677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:26.547773 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:26.548250 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:26.548253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:26.548254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:26.548442 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:26.548630 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:26.549190 (+ 560us) server_negotiation.cc:299] Negotiation successful
0807 06:22:26.549294 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 06:22:26.549681 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 18482
I20260807 06:22:26.549829 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance
I20260807 06:22:26.550117 16376 external_mini_cluster.cc:1428] Setting key 3445c28502459f39bed94b14ab18b0f1
I20260807 06:22:26.550472 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:26.533448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39757 (local address 127.15.254.3:36031)
0807 06:22:26.534019 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:26.534090 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:26.535444 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:26.536289 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:26.536303 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:26.536955 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:26.537811 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:26.537840 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.538847 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.538850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:26.539876 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:26.539887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.540455 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.540460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:26.540698 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:26.541410 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:26.541432 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:26.544925 (+ 3493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:26.547160 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:26.547166 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:26.547181 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:26.547531 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:26.547985 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:26.547988 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:26.547989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:26.548144 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:26.548623 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:26.548629 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:26.548934 (+ 305us) client_negotiation.cc:769] Sending connection context
0807 06:22:26.549134 (+ 200us) client_negotiation.cc:240] Negotiation successful
0807 06:22:26.549419 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":104,"threads_started":1}
I20260807 06:22:26.551765 18677 heartbeater.cc:347] Connected to a master server at 127.15.254.62:39757
I20260807 06:22:26.551952 18677 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:26.552639 18677 heartbeater.cc:511] Master 127.15.254.62:39757 requested a full tablet report, sending...
I20260807 06:22:26.553787 17982 ts_manager.cc:194] Registered new tserver with Master: 588e6c5989f04539b084ee778e9ed8a7 (127.15.254.3:45095)
I20260807 06:22:26.554643 17982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.3@KRBTEST.COM'} at 127.15.254.3:36031
I20260807 06:22:26.561615 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:26.554129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:44847 (local address 127.15.254.62:39757)
0807 06:22:26.554271 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:26.554276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:26.554443 (+ 167us) server_negotiation.cc:407] Connection header received
0807 06:22:26.554530 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:26.554534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:26.554581 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:26.554682 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:26.555120 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.555867 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.556815 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.557056 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.557141 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:26.558279 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:26.558282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:26.558283 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:26.558305 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:26.559709 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:26.560297 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:26.560300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:26.560301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:26.560386 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:26.560822 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:26.560824 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:26.560826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:26.560999 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:26.561084 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:26.561364 (+ 280us) server_negotiation.cc:299] Negotiation successful
0807 06:22:26.561461 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 06:22:26.561626 18685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:26.554071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39757 (local address 127.15.254.3:44847)
0807 06:22:26.554347 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:26.554363 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:26.554465 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:26.554702 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:26.554704 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:26.554879 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:26.554976 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:26.554981 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.556023 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.556034 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:26.556694 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:26.556704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.557163 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.557166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:26.557241 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:26.557810 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:26.557822 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:26.558152 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:26.559903 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:26.559906 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:26.559907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:26.560144 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:26.560612 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:26.560615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:26.560616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:26.560716 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:26.561102 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:26.561105 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:26.561199 (+ 94us) client_negotiation.cc:769] Sending connection context
0807 06:22:26.561305 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 06:22:26.561440 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":107,"threads_started":1}
I20260807 06:22:26.565160 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 06:22:26 dist-test-slave-44d2 krb5kdc[17916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083746, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 06:22:26 dist-test-slave-44d2 krb5kdc[17916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083746, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:26.593308 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:26.584922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42970 (local address 127.15.254.62:39757)
0807 06:22:26.585064 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:26.585069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:26.585086 (+ 17us) server_negotiation.cc:407] Connection header received
0807 06:22:26.585132 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:26.585134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:26.585177 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:26.585250 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:26.586067 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.586706 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.587549 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.587782 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.587867 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:26.590349 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:26.590353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:26.590356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:26.590383 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:26.591716 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:26.592105 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:26.592107 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:26.592108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:26.592187 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:26.592441 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:26.592443 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:26.592444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:26.592741 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:26.592854 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:26.593082 (+ 228us) server_negotiation.cc:299] Negotiation successful
0807 06:22:26.593181 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 06:22:26.596247 17982 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:22:26.599355 17982 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:22:26.614236 18702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:26.608459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40234 (local address 127.15.254.2:40227)
0807 06:22:26.609032 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:26.609048 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:26.609075 (+ 27us) server_negotiation.cc:407] Connection header received
0807 06:22:26.609159 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:26.609165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:26.609414 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:26.609584 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:26.610054 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.611203 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.612887 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.613510 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.613648 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:26.613853 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:26.613966 (+ 113us) server_negotiation.cc:299] Negotiation successful
0807 06:22:26.614075 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":123,"threads_started":1}
I20260807 06:22:26.614239 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:26.608209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:40227 (local address 127.0.0.1:40234)
0807 06:22:26.608602 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:26.608648 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:26.608808 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:26.609770 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:26.609776 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:26.609801 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:26.609923 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:26.609929 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.611317 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.611322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:26.612752 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:26.612764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.613648 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.613652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:26.613766 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:26.613807 (+ 41us) client_negotiation.cc:769] Sending connection context
0807 06:22:26.613934 (+ 127us) client_negotiation.cc:240] Negotiation successful
0807 06:22:26.614067 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":28,"thread_start_us":105,"threads_started":1}
I20260807 06:22:26.615573 18705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:26.609025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:45095 (local address 127.0.0.1:37846)
0807 06:22:26.609875 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:26.609889 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:26.609998 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:26.610716 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:26.610719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:26.610734 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:26.610810 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:26.610813 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.612340 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.612344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:26.613569 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:26.613579 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.615087 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.615091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:26.615192 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:26.615220 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 06:22:26.615326 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 06:22:26.615424 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":1,"thread_start_us":116,"threads_started":1}
I20260807 06:22:26.616060 18703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:26.609387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:36583 (local address 127.0.0.1:60116)
0807 06:22:26.610961 (+ 1574us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:26.610975 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:26.611068 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:26.611638 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:26.611641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:26.611658 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:26.611738 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:26.611744 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.612986 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.612989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:26.614152 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:26.614164 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.615447 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.615451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:26.615552 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:26.615578 (+ 26us) client_negotiation.cc:769] Sending connection context
0807 06:22:26.615851 (+ 273us) client_negotiation.cc:240] Negotiation successful
0807 06:22:26.615930 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1511,"thread_start_us":106,"threads_started":1}
I20260807 06:22:26.616104 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:26.609573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60116 (local address 127.15.254.1:36583)
0807 06:22:26.609860 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:26.609866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:26.611080 (+ 1214us) server_negotiation.cc:407] Connection header received
0807 06:22:26.611165 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:26.611170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:26.611351 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:26.611489 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:26.611860 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.612870 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.614313 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.615330 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.615428 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:26.615738 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:26.615801 (+ 63us) server_negotiation.cc:299] Negotiation successful
0807 06:22:26.615954 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20260807 06:22:26.616552 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:26.609208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37846 (local address 127.15.254.3:45095)
0807 06:22:26.610264 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:26.610276 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:26.610304 (+ 28us) server_negotiation.cc:407] Connection header received
0807 06:22:26.610381 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:26.610386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:26.610581 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:26.610732 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:26.610937 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.612206 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.613718 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.614930 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.615882 (+ 952us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:26.616046 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:26.616113 (+ 67us) server_negotiation.cc:299] Negotiation successful
0807 06:22:26.616218 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":949,"thread_start_us":115,"threads_started":1}
I20260807 06:22:26.618090 18376 tablet_service.cc:1467] Processing CreateTablet for tablet 340c326b1aee418ca6011c4f0c8974cb (DEFAULT_TABLE table=test-table [id=6e3246b0e95041368fb3b3c5fcb14501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:26.619342 18376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 340c326b1aee418ca6011c4f0c8974cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:26.619302 18180 tablet_service.cc:1467] Processing CreateTablet for tablet 340c326b1aee418ca6011c4f0c8974cb (DEFAULT_TABLE table=test-table [id=6e3246b0e95041368fb3b3c5fcb14501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:26.620317 18180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 340c326b1aee418ca6011c4f0c8974cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:26.620544 18582 tablet_service.cc:1467] Processing CreateTablet for tablet 340c326b1aee418ca6011c4f0c8974cb (DEFAULT_TABLE table=test-table [id=6e3246b0e95041368fb3b3c5fcb14501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:26.621362 18582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 340c326b1aee418ca6011c4f0c8974cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:26.627667 18708 tablet_bootstrap.cc:492] T 340c326b1aee418ca6011c4f0c8974cb P f0654852d6404e6488ebce8e50bf9943: Bootstrap starting.
I20260807 06:22:26.628718 18711 tablet_bootstrap.cc:492] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7: Bootstrap starting.
I20260807 06:22:26.630736 18708 tablet_bootstrap.cc:654] T 340c326b1aee418ca6011c4f0c8974cb P f0654852d6404e6488ebce8e50bf9943: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:26.630821 18711 tablet_bootstrap.cc:654] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:26.631723 18711 log.cc:824] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:26.631723 18708 log.cc:824] T 340c326b1aee418ca6011c4f0c8974cb P f0654852d6404e6488ebce8e50bf9943: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:26.632375 18709 tablet_bootstrap.cc:492] T 340c326b1aee418ca6011c4f0c8974cb P 3a95e762028341a59c1f85991a16607a: Bootstrap starting.
I20260807 06:22:26.633920 18708 tablet_bootstrap.cc:492] T 340c326b1aee418ca6011c4f0c8974cb P f0654852d6404e6488ebce8e50bf9943: No bootstrap required, opened a new log
I20260807 06:22:26.633920 18711 tablet_bootstrap.cc:492] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7: No bootstrap required, opened a new log
I20260807 06:22:26.634128 18711 ts_tablet_manager.cc:1397] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 06:22:26.634128 18708 ts_tablet_manager.cc:1397] T 340c326b1aee418ca6011c4f0c8974cb P f0654852d6404e6488ebce8e50bf9943: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260807 06:22:26.634357 18709 tablet_bootstrap.cc:654] T 340c326b1aee418ca6011c4f0c8974cb P 3a95e762028341a59c1f85991a16607a: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:26.635046 18709 log.cc:824] T 340c326b1aee418ca6011c4f0c8974cb P 3a95e762028341a59c1f85991a16607a: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:26.637316 18709 tablet_bootstrap.cc:492] T 340c326b1aee418ca6011c4f0c8974cb P 3a95e762028341a59c1f85991a16607a: No bootstrap required, opened a new log
I20260807 06:22:26.637516 18709 ts_tablet_manager.cc:1397] T 340c326b1aee418ca6011c4f0c8974cb P 3a95e762028341a59c1f85991a16607a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260807 06:22:26.638855 18708 raft_consensus.cc:348] T 340c326b1aee418ca6011c4f0c8974cb P f0654852d6404e6488ebce8e50bf9943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } } peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } }
I20260807 06:22:26.638854 18711 raft_consensus.cc:348] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } } peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } }
I20260807 06:22:26.639144 18708 raft_consensus.cc:374] T 340c326b1aee418ca6011c4f0c8974cb P f0654852d6404e6488ebce8e50bf9943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:26.639144 18711 raft_consensus.cc:374] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:26.639209 18708 raft_consensus.cc:729] T 340c326b1aee418ca6011c4f0c8974cb P f0654852d6404e6488ebce8e50bf9943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0654852d6404e6488ebce8e50bf9943, State: Initialized, Role: FOLLOWER
I20260807 06:22:26.639209 18711 raft_consensus.cc:729] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588e6c5989f04539b084ee778e9ed8a7, State: Initialized, Role: FOLLOWER
I20260807 06:22:26.639724 18711 consensus_queue.cc:260] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } } peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } }
I20260807 06:22:26.639724 18708 consensus_queue.cc:260] T 340c326b1aee418ca6011c4f0c8974cb P f0654852d6404e6488ebce8e50bf9943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } } peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } }
I20260807 06:22:26.640533 18265 heartbeater.cc:503] Master 127.15.254.62:39757 was elected leader, sending a full tablet report...
I20260807 06:22:26.640825 18711 ts_tablet_manager.cc:1428] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260807 06:22:26.640980 18677 heartbeater.cc:503] Master 127.15.254.62:39757 was elected leader, sending a full tablet report...
I20260807 06:22:26.640825 18708 ts_tablet_manager.cc:1428] T 340c326b1aee418ca6011c4f0c8974cb P f0654852d6404e6488ebce8e50bf9943: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260807 06:22:26.642052 18709 raft_consensus.cc:348] T 340c326b1aee418ca6011c4f0c8974cb P 3a95e762028341a59c1f85991a16607a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } } peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } }
I20260807 06:22:26.642310 18709 raft_consensus.cc:374] T 340c326b1aee418ca6011c4f0c8974cb P 3a95e762028341a59c1f85991a16607a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:26.642379 18709 raft_consensus.cc:729] T 340c326b1aee418ca6011c4f0c8974cb P 3a95e762028341a59c1f85991a16607a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a95e762028341a59c1f85991a16607a, State: Initialized, Role: FOLLOWER
I20260807 06:22:26.642798 18709 consensus_queue.cc:260] T 340c326b1aee418ca6011c4f0c8974cb P 3a95e762028341a59c1f85991a16607a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } } peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } }
I20260807 06:22:26.643534 18469 heartbeater.cc:503] Master 127.15.254.62:39757 was elected leader, sending a full tablet report...
I20260807 06:22:26.643874 18709 ts_tablet_manager.cc:1428] T 340c326b1aee418ca6011c4f0c8974cb P 3a95e762028341a59c1f85991a16607a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260807 06:22:26.797885 18679 tablet.cc:2369] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 06:22:26.813052 18470 tablet.cc:2369] T 340c326b1aee418ca6011c4f0c8974cb P 3a95e762028341a59c1f85991a16607a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:22:26.830616 18718 raft_consensus.cc:482] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:22:26.830974 18718 raft_consensus.cc:504] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } } peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } }
I20260807 06:22:26.832239 18718 leader_election.cc:288] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a95e762028341a59c1f85991a16607a (127.15.254.2:40227), f0654852d6404e6488ebce8e50bf9943 (127.15.254.1:36583)
W20260807 06:22:26.832782 18266 tablet.cc:2369] T 340c326b1aee418ca6011c4f0c8974cb P f0654852d6404e6488ebce8e50bf9943: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:22:26.836328 18702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:26.832703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:60139 (local address 127.15.254.2:40227)
0807 06:22:26.832863 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:26.832868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:26.832916 (+ 48us) server_negotiation.cc:407] Connection header received
0807 06:22:26.833102 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:26.833105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:26.833157 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:26.833249 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:26.833676 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.834218 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.835324 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.835734 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.835882 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:26.835945 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:26.836132 (+ 187us) server_negotiation.cc:299] Negotiation successful
0807 06:22:26.836209 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 06:22:26.836328 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:26.832685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:40227 (local address 127.15.254.3:60139)
0807 06:22:26.832838 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:26.832856 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:26.832981 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:26.833392 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:26.833396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:26.833414 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:26.833543 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:26.833549 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.834354 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.834357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:26.835186 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:26.835195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.835866 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.835870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:26.835988 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:26.836016 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 06:22:26.836115 (+ 99us) client_negotiation.cc:240] Negotiation successful
0807 06:22:26.836187 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 06:22:26.837173 18720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:26.833020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:36583 (local address 127.15.254.3:35739)
0807 06:22:26.833464 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:26.833478 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:26.833604 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:26.833873 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:26.833875 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:26.833891 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:26.834345 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:26.834353 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.835107 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.835114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:26.835871 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:26.835881 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.836720 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.836724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:26.836825 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:26.836857 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 06:22:26.836922 (+ 65us) client_negotiation.cc:240] Negotiation successful
0807 06:22:26.837036 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":136,"threads_started":1}
I20260807 06:22:26.837180 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:26.833322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:35739 (local address 127.15.254.1:36583)
0807 06:22:26.833454 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:26.833459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:26.833563 (+ 104us) server_negotiation.cc:407] Connection header received
0807 06:22:26.833727 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:26.833731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:26.833772 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:26.833861 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:26.834482 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.834975 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.836012 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.836554 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.836687 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:26.836733 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:26.836973 (+ 240us) server_negotiation.cc:299] Negotiation successful
0807 06:22:26.837053 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 06:22:26.837121 18402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340c326b1aee418ca6011c4f0c8974cb" candidate_uuid: "588e6c5989f04539b084ee778e9ed8a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a95e762028341a59c1f85991a16607a" is_pre_election: true
I20260807 06:22:26.837495 18402 raft_consensus.cc:2454] T 340c326b1aee418ca6011c4f0c8974cb P 3a95e762028341a59c1f85991a16607a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588e6c5989f04539b084ee778e9ed8a7 in term 0.
I20260807 06:22:26.837831 18206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340c326b1aee418ca6011c4f0c8974cb" candidate_uuid: "588e6c5989f04539b084ee778e9ed8a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0654852d6404e6488ebce8e50bf9943" is_pre_election: true
I20260807 06:22:26.838024 18504 leader_election.cc:302] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a95e762028341a59c1f85991a16607a, 588e6c5989f04539b084ee778e9ed8a7; no voters:
I20260807 06:22:26.838135 18206 raft_consensus.cc:2454] T 340c326b1aee418ca6011c4f0c8974cb P f0654852d6404e6488ebce8e50bf9943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588e6c5989f04539b084ee778e9ed8a7 in term 0.
I20260807 06:22:26.838310 18718 raft_consensus.cc:2781] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:22:26.838435 18718 raft_consensus.cc:482] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:22:26.838503 18718 raft_consensus.cc:3037] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:26.839665 18718 raft_consensus.cc:504] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } } peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } }
I20260807 06:22:26.840049 18718 leader_election.cc:288] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7 [CANDIDATE]: Term 1 election: Requested vote from peers 3a95e762028341a59c1f85991a16607a (127.15.254.2:40227), f0654852d6404e6488ebce8e50bf9943 (127.15.254.1:36583)
I20260807 06:22:26.840574 18402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340c326b1aee418ca6011c4f0c8974cb" candidate_uuid: "588e6c5989f04539b084ee778e9ed8a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a95e762028341a59c1f85991a16607a"
I20260807 06:22:26.840689 18206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340c326b1aee418ca6011c4f0c8974cb" candidate_uuid: "588e6c5989f04539b084ee778e9ed8a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0654852d6404e6488ebce8e50bf9943"
I20260807 06:22:26.840727 18402 raft_consensus.cc:3037] T 340c326b1aee418ca6011c4f0c8974cb P 3a95e762028341a59c1f85991a16607a [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:26.840878 18206 raft_consensus.cc:3037] T 340c326b1aee418ca6011c4f0c8974cb P f0654852d6404e6488ebce8e50bf9943 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:26.842078 18402 raft_consensus.cc:2454] T 340c326b1aee418ca6011c4f0c8974cb P 3a95e762028341a59c1f85991a16607a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588e6c5989f04539b084ee778e9ed8a7 in term 1.
I20260807 06:22:26.842167 18206 raft_consensus.cc:2454] T 340c326b1aee418ca6011c4f0c8974cb P f0654852d6404e6488ebce8e50bf9943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588e6c5989f04539b084ee778e9ed8a7 in term 1.
I20260807 06:22:26.842480 18504 leader_election.cc:302] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a95e762028341a59c1f85991a16607a, 588e6c5989f04539b084ee778e9ed8a7; no voters:
I20260807 06:22:26.842677 18718 raft_consensus.cc:2781] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:26.842908 18718 raft_consensus.cc:686] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7 [term 1 LEADER]: Becoming Leader. State: Replica: 588e6c5989f04539b084ee778e9ed8a7, State: Running, Role: LEADER
I20260807 06:22:26.843276 18718 consensus_queue.cc:237] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } } peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } }
I20260807 06:22:26.847015 17982 catalog_manager.cc:5495] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588e6c5989f04539b084ee778e9ed8a7 (127.15.254.3). New cstate: current_term: 1 leader_uuid: "588e6c5989f04539b084ee778e9ed8a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } health_report { overall_health: UNKNOWN } } }
I20260807 06:22:26.918883 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:26.914816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37852 (local address 127.15.254.3:45095)
0807 06:22:26.914959 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:26.914963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:26.914978 (+ 15us) server_negotiation.cc:407] Connection header received
0807 06:22:26.915167 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:26.915171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:26.915304 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:26.915395 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:22:26.915701 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.916216 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.917384 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:26.917618 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:26.917719 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:26.918060 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:22:26.918528 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:22:26.918622 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:26.918705 (+ 83us) server_negotiation.cc:299] Negotiation successful
0807 06:22:26.918763 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 06:22:26.925479 18402 raft_consensus.cc:1261] T 340c326b1aee418ca6011c4f0c8974cb P 3a95e762028341a59c1f85991a16607a [term 1 FOLLOWER]: Refusing update from remote peer 588e6c5989f04539b084ee778e9ed8a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:22:26.925518 18206 raft_consensus.cc:1261] T 340c326b1aee418ca6011c4f0c8974cb P f0654852d6404e6488ebce8e50bf9943 [term 1 FOLLOWER]: Refusing update from remote peer 588e6c5989f04539b084ee778e9ed8a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:22:26.926268 18721 consensus_queue.cc:1035] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:22:26.926486 18718 consensus_queue.cc:1035] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:22:26.935266 18727 mvcc.cc:205] Tried to move back new op lower bound from 7315799027398393856 to 7315799027072946176. Current Snapshot: MvccSnapshot[applied={T|T < 7315799027398393856}]
I20260807 06:22:26.949713 17981 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:22:26.949996 17981 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:22:26.953064 17981 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 099cd300fbd84bcfbcd9554503e7a55f: Sending DeleteTablet for 3 replicas of tablet 340c326b1aee418ca6011c4f0c8974cb
I20260807 06:22:26.954252 18376 tablet_service.cc:1514] Processing DeleteTablet for tablet 340c326b1aee418ca6011c4f0c8974cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:26 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:40234
I20260807 06:22:26.954409 18582 tablet_service.cc:1514] Processing DeleteTablet for tablet 340c326b1aee418ca6011c4f0c8974cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:26 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:37846
I20260807 06:22:26.954480 18180 tablet_service.cc:1514] Processing DeleteTablet for tablet 340c326b1aee418ca6011c4f0c8974cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:26 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:60116
I20260807 06:22:26.954967 17981 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 06:22:26.955097 18743 tablet_replica.cc:330] T 340c326b1aee418ca6011c4f0c8974cb P f0654852d6404e6488ebce8e50bf9943: stopping tablet replica
W20260807 06:22:26.955432 17981 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:22:26.955478 18743 raft_consensus.cc:2229] T 340c326b1aee418ca6011c4f0c8974cb P f0654852d6404e6488ebce8e50bf9943 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:22:26.955470 18742 tablet_replica.cc:330] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7: stopping tablet replica
I20260807 06:22:26.956022 18742 raft_consensus.cc:2229] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:22:26.956081 18743 raft_consensus.cc:2258] T 340c326b1aee418ca6011c4f0c8974cb P f0654852d6404e6488ebce8e50bf9943 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:26.956763 18741 tablet_replica.cc:330] T 340c326b1aee418ca6011c4f0c8974cb P 3a95e762028341a59c1f85991a16607a: stopping tablet replica
I20260807 06:22:26.957105 18741 raft_consensus.cc:2229] T 340c326b1aee418ca6011c4f0c8974cb P 3a95e762028341a59c1f85991a16607a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:22:26.957444 18741 raft_consensus.cc:2258] T 340c326b1aee418ca6011c4f0c8974cb P 3a95e762028341a59c1f85991a16607a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:26.957505 18742 raft_consensus.cc:2258] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:26.959179 18741 ts_tablet_manager.cc:1905] T 340c326b1aee418ca6011c4f0c8974cb P 3a95e762028341a59c1f85991a16607a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:22:26.961812 18376 tablet_service.cc:1467] Processing CreateTablet for tablet 24845fb791d74e909bb6a7b1596e9fbe (DEFAULT_TABLE table=test-table [id=4e46f5fe4a9e48648ac3bb4ad4afc7e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:26.962040 18180 tablet_service.cc:1467] Processing CreateTablet for tablet 24845fb791d74e909bb6a7b1596e9fbe (DEFAULT_TABLE table=test-table [id=4e46f5fe4a9e48648ac3bb4ad4afc7e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:26.962133 18376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24845fb791d74e909bb6a7b1596e9fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:26.962090 18582 tablet_service.cc:1467] Processing CreateTablet for tablet 24845fb791d74e909bb6a7b1596e9fbe (DEFAULT_TABLE table=test-table [id=4e46f5fe4a9e48648ac3bb4ad4afc7e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:26.962296 18180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24845fb791d74e909bb6a7b1596e9fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:26.962369 18741 ts_tablet_manager.cc:1918] T 340c326b1aee418ca6011c4f0c8974cb P 3a95e762028341a59c1f85991a16607a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:22:26.962405 18582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24845fb791d74e909bb6a7b1596e9fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:26.962468 18741 log.cc:1196] T 340c326b1aee418ca6011c4f0c8974cb P 3a95e762028341a59c1f85991a16607a: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/340c326b1aee418ca6011c4f0c8974cb
I20260807 06:22:26.962824 18741 ts_tablet_manager.cc:1939] T 340c326b1aee418ca6011c4f0c8974cb P 3a95e762028341a59c1f85991a16607a: Deleting consensus metadata
I20260807 06:22:26.964615 18709 tablet_bootstrap.cc:492] T 24845fb791d74e909bb6a7b1596e9fbe P 3a95e762028341a59c1f85991a16607a: Bootstrap starting.
I20260807 06:22:26.965233 17958 catalog_manager.cc:4841] TS 3a95e762028341a59c1f85991a16607a (127.15.254.2:40227): tablet 340c326b1aee418ca6011c4f0c8974cb (table test-table [id=6e3246b0e95041368fb3b3c5fcb14501]) successfully deleted
I20260807 06:22:26.965692 18708 tablet_bootstrap.cc:492] T 24845fb791d74e909bb6a7b1596e9fbe P f0654852d6404e6488ebce8e50bf9943: Bootstrap starting.
I20260807 06:22:26.965802 18709 tablet_bootstrap.cc:654] T 24845fb791d74e909bb6a7b1596e9fbe P 3a95e762028341a59c1f85991a16607a: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:26.966809 18711 tablet_bootstrap.cc:492] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7: Bootstrap starting.
I20260807 06:22:26.966959 18708 tablet_bootstrap.cc:654] T 24845fb791d74e909bb6a7b1596e9fbe P f0654852d6404e6488ebce8e50bf9943: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:26.968158 18709 tablet_bootstrap.cc:492] T 24845fb791d74e909bb6a7b1596e9fbe P 3a95e762028341a59c1f85991a16607a: No bootstrap required, opened a new log
I20260807 06:22:26.968191 18711 tablet_bootstrap.cc:654] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:26.968250 18709 ts_tablet_manager.cc:1397] T 24845fb791d74e909bb6a7b1596e9fbe P 3a95e762028341a59c1f85991a16607a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260807 06:22:26.968885 18742 ts_tablet_manager.cc:1905] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:22:26.968842 18709 raft_consensus.cc:348] T 24845fb791d74e909bb6a7b1596e9fbe P 3a95e762028341a59c1f85991a16607a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } }
I20260807 06:22:26.968982 18709 raft_consensus.cc:374] T 24845fb791d74e909bb6a7b1596e9fbe P 3a95e762028341a59c1f85991a16607a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:26.969022 18709 raft_consensus.cc:729] T 24845fb791d74e909bb6a7b1596e9fbe P 3a95e762028341a59c1f85991a16607a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a95e762028341a59c1f85991a16607a, State: Initialized, Role: FOLLOWER
I20260807 06:22:26.969172 18709 consensus_queue.cc:260] T 24845fb791d74e909bb6a7b1596e9fbe P 3a95e762028341a59c1f85991a16607a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } }
I20260807 06:22:26.969489 18709 ts_tablet_manager.cc:1428] T 24845fb791d74e909bb6a7b1596e9fbe P 3a95e762028341a59c1f85991a16607a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260807 06:22:26.971874 18742 ts_tablet_manager.cc:1918] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:22:26.971983 18742 log.cc:1196] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/340c326b1aee418ca6011c4f0c8974cb
I20260807 06:22:26.972304 18742 ts_tablet_manager.cc:1939] T 340c326b1aee418ca6011c4f0c8974cb P 588e6c5989f04539b084ee778e9ed8a7: Deleting consensus metadata
I20260807 06:22:26.973490 18708 tablet_bootstrap.cc:492] T 24845fb791d74e909bb6a7b1596e9fbe P f0654852d6404e6488ebce8e50bf9943: No bootstrap required, opened a new log
I20260807 06:22:26.973603 18708 ts_tablet_manager.cc:1397] T 24845fb791d74e909bb6a7b1596e9fbe P f0654852d6404e6488ebce8e50bf9943: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260807 06:22:26.973826 17958 catalog_manager.cc:4841] TS 588e6c5989f04539b084ee778e9ed8a7 (127.15.254.3:45095): tablet 340c326b1aee418ca6011c4f0c8974cb (table test-table [id=6e3246b0e95041368fb3b3c5fcb14501]) successfully deleted
I20260807 06:22:26.973974 18743 ts_tablet_manager.cc:1905] T 340c326b1aee418ca6011c4f0c8974cb P f0654852d6404e6488ebce8e50bf9943: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:22:26.974182 18708 raft_consensus.cc:348] T 24845fb791d74e909bb6a7b1596e9fbe P f0654852d6404e6488ebce8e50bf9943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } }
I20260807 06:22:26.974290 18708 raft_consensus.cc:374] T 24845fb791d74e909bb6a7b1596e9fbe P f0654852d6404e6488ebce8e50bf9943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:26.974314 18708 raft_consensus.cc:729] T 24845fb791d74e909bb6a7b1596e9fbe P f0654852d6404e6488ebce8e50bf9943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0654852d6404e6488ebce8e50bf9943, State: Initialized, Role: FOLLOWER
I20260807 06:22:26.974489 18708 consensus_queue.cc:260] T 24845fb791d74e909bb6a7b1596e9fbe P f0654852d6404e6488ebce8e50bf9943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } }
I20260807 06:22:26.974977 18708 ts_tablet_manager.cc:1428] T 24845fb791d74e909bb6a7b1596e9fbe P f0654852d6404e6488ebce8e50bf9943: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260807 06:22:26.977856 18711 tablet_bootstrap.cc:492] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7: No bootstrap required, opened a new log
I20260807 06:22:26.977977 18711 ts_tablet_manager.cc:1397] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20260807 06:22:26.978278 18743 ts_tablet_manager.cc:1918] T 340c326b1aee418ca6011c4f0c8974cb P f0654852d6404e6488ebce8e50bf9943: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:22:26.978399 18743 log.cc:1196] T 340c326b1aee418ca6011c4f0c8974cb P f0654852d6404e6488ebce8e50bf9943: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-0/wal/wals/340c326b1aee418ca6011c4f0c8974cb
I20260807 06:22:26.978674 18711 raft_consensus.cc:348] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } }
I20260807 06:22:26.978770 18743 ts_tablet_manager.cc:1939] T 340c326b1aee418ca6011c4f0c8974cb P f0654852d6404e6488ebce8e50bf9943: Deleting consensus metadata
I20260807 06:22:26.978824 18711 raft_consensus.cc:374] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:26.978870 18711 raft_consensus.cc:729] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588e6c5989f04539b084ee778e9ed8a7, State: Initialized, Role: FOLLOWER
I20260807 06:22:26.979034 18711 consensus_queue.cc:260] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } }
I20260807 06:22:26.979321 18711 ts_tablet_manager.cc:1428] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260807 06:22:26.980233 17958 catalog_manager.cc:4841] TS f0654852d6404e6488ebce8e50bf9943 (127.15.254.1:36583): tablet 340c326b1aee418ca6011c4f0c8974cb (table test-table [id=6e3246b0e95041368fb3b3c5fcb14501]) successfully deleted
I20260807 06:22:27.146698 18730 raft_consensus.cc:482] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:22:27.146869 18730 raft_consensus.cc:504] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } }
I20260807 06:22:27.147385 18730 leader_election.cc:288] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0654852d6404e6488ebce8e50bf9943 (127.15.254.1:36583), 3a95e762028341a59c1f85991a16607a (127.15.254.2:40227)
I20260807 06:22:27.147922 18206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24845fb791d74e909bb6a7b1596e9fbe" candidate_uuid: "588e6c5989f04539b084ee778e9ed8a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0654852d6404e6488ebce8e50bf9943" is_pre_election: true
I20260807 06:22:27.147922 18402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24845fb791d74e909bb6a7b1596e9fbe" candidate_uuid: "588e6c5989f04539b084ee778e9ed8a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a95e762028341a59c1f85991a16607a" is_pre_election: true
I20260807 06:22:27.148078 18402 raft_consensus.cc:2454] T 24845fb791d74e909bb6a7b1596e9fbe P 3a95e762028341a59c1f85991a16607a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588e6c5989f04539b084ee778e9ed8a7 in term 0.
I20260807 06:22:27.148078 18206 raft_consensus.cc:2454] T 24845fb791d74e909bb6a7b1596e9fbe P f0654852d6404e6488ebce8e50bf9943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588e6c5989f04539b084ee778e9ed8a7 in term 0.
I20260807 06:22:27.148471 18504 leader_election.cc:302] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a95e762028341a59c1f85991a16607a, 588e6c5989f04539b084ee778e9ed8a7; no voters:
I20260807 06:22:27.148710 18730 raft_consensus.cc:2781] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:22:27.148767 18730 raft_consensus.cc:482] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:22:27.148808 18730 raft_consensus.cc:3037] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:27.150348 18730 raft_consensus.cc:504] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } }
I20260807 06:22:27.150681 18730 leader_election.cc:288] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7 [CANDIDATE]: Term 1 election: Requested vote from peers f0654852d6404e6488ebce8e50bf9943 (127.15.254.1:36583), 3a95e762028341a59c1f85991a16607a (127.15.254.2:40227)
I20260807 06:22:27.151139 18402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24845fb791d74e909bb6a7b1596e9fbe" candidate_uuid: "588e6c5989f04539b084ee778e9ed8a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a95e762028341a59c1f85991a16607a"
I20260807 06:22:27.151126 18206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24845fb791d74e909bb6a7b1596e9fbe" candidate_uuid: "588e6c5989f04539b084ee778e9ed8a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0654852d6404e6488ebce8e50bf9943"
I20260807 06:22:27.151248 18206 raft_consensus.cc:3037] T 24845fb791d74e909bb6a7b1596e9fbe P f0654852d6404e6488ebce8e50bf9943 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:27.151247 18402 raft_consensus.cc:3037] T 24845fb791d74e909bb6a7b1596e9fbe P 3a95e762028341a59c1f85991a16607a [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:27.152511 18402 raft_consensus.cc:2454] T 24845fb791d74e909bb6a7b1596e9fbe P 3a95e762028341a59c1f85991a16607a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588e6c5989f04539b084ee778e9ed8a7 in term 1.
I20260807 06:22:27.152511 18206 raft_consensus.cc:2454] T 24845fb791d74e909bb6a7b1596e9fbe P f0654852d6404e6488ebce8e50bf9943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588e6c5989f04539b084ee778e9ed8a7 in term 1.
I20260807 06:22:27.152871 18504 leader_election.cc:302] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a95e762028341a59c1f85991a16607a, 588e6c5989f04539b084ee778e9ed8a7; no voters:
I20260807 06:22:27.153138 18730 raft_consensus.cc:2781] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:27.153260 18730 raft_consensus.cc:686] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7 [term 1 LEADER]: Becoming Leader. State: Replica: 588e6c5989f04539b084ee778e9ed8a7, State: Running, Role: LEADER
I20260807 06:22:27.153414 18730 consensus_queue.cc:237] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } }
I20260807 06:22:27.155118 17981 catalog_manager.cc:5495] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588e6c5989f04539b084ee778e9ed8a7 (127.15.254.3). New cstate: current_term: 1 leader_uuid: "588e6c5989f04539b084ee778e9ed8a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } health_report { overall_health: HEALTHY } } }
Aug 07 06:22:27 dist-test-slave-44d2 krb5kdc[17916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083745, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:27.203799 18759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:27.194453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42978 (local address 127.15.254.62:39757)
0807 06:22:27.194876 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:27.194888 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:27.194905 (+ 17us) server_negotiation.cc:407] Connection header received
0807 06:22:27.195091 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:27.195094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:27.195143 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:27.195243 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:27.195942 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.196566 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.197387 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.197618 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.197733 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:27.200089 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:27.200093 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:27.200095 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:27.200124 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:27.201836 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:27.202318 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:27.202324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:27.202326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:27.202411 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:27.202728 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:27.202731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:27.202733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:27.203008 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:27.203129 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:27.203441 (+ 312us) server_negotiation.cc:299] Negotiation successful
0807 06:22:27.203656 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":106,"server-negotiator.queue_time_us":362,"thread_start_us":172,"threads_started":1}
I20260807 06:22:27.203799 18758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:27.194439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39757 (local address 127.0.0.1:42978)
0807 06:22:27.194829 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:27.194847 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:27.194977 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:27.195267 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:27.195270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:27.195716 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:27.195815 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:27.195822 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.196701 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.196705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:27.197262 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:27.197270 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.197735 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.197738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:27.197809 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:27.198444 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:27.198457 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:27.199946 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:27.201948 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:27.201961 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:27.201968 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:27.202213 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:27.202550 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:27.202553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:27.202554 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:27.202634 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:27.203130 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:27.203138 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:27.203269 (+ 131us) client_negotiation.cc:769] Sending connection context
0807 06:22:27.203393 (+ 124us) client_negotiation.cc:240] Negotiation successful
0807 06:22:27.203590 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":133,"threads_started":1}
I20260807 06:22:27.208587 17982 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:42978:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 06:22:27.209501 17982 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:22:27.216498 18180 tablet_service.cc:1467] Processing CreateTablet for tablet b674d97a62f042deb30e4ec72a7572d8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=884a789ed6374f7eb4490aef23bff4d9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 06:22:27.217028 18180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b674d97a62f042deb30e4ec72a7572d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:27.216498 18376 tablet_service.cc:1467] Processing CreateTablet for tablet b674d97a62f042deb30e4ec72a7572d8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=884a789ed6374f7eb4490aef23bff4d9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 06:22:27.218412 18376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b674d97a62f042deb30e4ec72a7572d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:27.220988 18582 tablet_service.cc:1467] Processing CreateTablet for tablet b674d97a62f042deb30e4ec72a7572d8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=884a789ed6374f7eb4490aef23bff4d9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 06:22:27.221376 18582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b674d97a62f042deb30e4ec72a7572d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:27.221712 18709 tablet_bootstrap.cc:492] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a: Bootstrap starting.
I20260807 06:22:27.222800 18709 tablet_bootstrap.cc:654] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:27.224184 18708 tablet_bootstrap.cc:492] T b674d97a62f042deb30e4ec72a7572d8 P f0654852d6404e6488ebce8e50bf9943: Bootstrap starting.
I20260807 06:22:27.225148 18708 tablet_bootstrap.cc:654] T b674d97a62f042deb30e4ec72a7572d8 P f0654852d6404e6488ebce8e50bf9943: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:27.226719 18711 tablet_bootstrap.cc:492] T b674d97a62f042deb30e4ec72a7572d8 P 588e6c5989f04539b084ee778e9ed8a7: Bootstrap starting.
I20260807 06:22:27.227411 18708 tablet_bootstrap.cc:492] T b674d97a62f042deb30e4ec72a7572d8 P f0654852d6404e6488ebce8e50bf9943: No bootstrap required, opened a new log
I20260807 06:22:27.227519 18708 ts_tablet_manager.cc:1397] T b674d97a62f042deb30e4ec72a7572d8 P f0654852d6404e6488ebce8e50bf9943: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 06:22:27.227671 18711 tablet_bootstrap.cc:654] T b674d97a62f042deb30e4ec72a7572d8 P 588e6c5989f04539b084ee778e9ed8a7: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:27.228031 18708 raft_consensus.cc:348] T b674d97a62f042deb30e4ec72a7572d8 P f0654852d6404e6488ebce8e50bf9943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } }
I20260807 06:22:27.228173 18708 raft_consensus.cc:374] T b674d97a62f042deb30e4ec72a7572d8 P f0654852d6404e6488ebce8e50bf9943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:27.228212 18708 raft_consensus.cc:729] T b674d97a62f042deb30e4ec72a7572d8 P f0654852d6404e6488ebce8e50bf9943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0654852d6404e6488ebce8e50bf9943, State: Initialized, Role: FOLLOWER
I20260807 06:22:27.229786 18709 tablet_bootstrap.cc:492] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a: No bootstrap required, opened a new log
I20260807 06:22:27.229882 18709 ts_tablet_manager.cc:1397] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260807 06:22:27.229825 18708 consensus_queue.cc:260] T b674d97a62f042deb30e4ec72a7572d8 P f0654852d6404e6488ebce8e50bf9943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } }
I20260807 06:22:27.230157 18717 tablet_replica.cc:439] T b674d97a62f042deb30e4ec72a7572d8 P f0654852d6404e6488ebce8e50bf9943: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } } }
I20260807 06:22:27.230305 18717 tablet_replica.cc:442] T b674d97a62f042deb30e4ec72a7572d8 P f0654852d6404e6488ebce8e50bf9943: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:22:27.230439 18709 raft_consensus.cc:348] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } }
I20260807 06:22:27.230547 18709 raft_consensus.cc:374] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:27.230581 18709 raft_consensus.cc:729] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a95e762028341a59c1f85991a16607a, State: Initialized, Role: FOLLOWER
I20260807 06:22:27.230700 18709 consensus_queue.cc:260] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } }
I20260807 06:22:27.230983 18709 ts_tablet_manager.cc:1428] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 06:22:27.231316 18719 tablet_replica.cc:439] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } } }
I20260807 06:22:27.231446 18719 tablet_replica.cc:442] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:22:27.232290 18711 tablet_bootstrap.cc:492] T b674d97a62f042deb30e4ec72a7572d8 P 588e6c5989f04539b084ee778e9ed8a7: No bootstrap required, opened a new log
I20260807 06:22:27.232385 18711 ts_tablet_manager.cc:1397] T b674d97a62f042deb30e4ec72a7572d8 P 588e6c5989f04539b084ee778e9ed8a7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260807 06:22:27.233039 18711 raft_consensus.cc:348] T b674d97a62f042deb30e4ec72a7572d8 P 588e6c5989f04539b084ee778e9ed8a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } }
I20260807 06:22:27.233175 18711 raft_consensus.cc:374] T b674d97a62f042deb30e4ec72a7572d8 P 588e6c5989f04539b084ee778e9ed8a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:27.233212 18711 raft_consensus.cc:729] T b674d97a62f042deb30e4ec72a7572d8 P 588e6c5989f04539b084ee778e9ed8a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588e6c5989f04539b084ee778e9ed8a7, State: Initialized, Role: FOLLOWER
I20260807 06:22:27.233350 18711 consensus_queue.cc:260] T b674d97a62f042deb30e4ec72a7572d8 P 588e6c5989f04539b084ee778e9ed8a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } }
I20260807 06:22:27.233639 18711 ts_tablet_manager.cc:1428] T b674d97a62f042deb30e4ec72a7572d8 P 588e6c5989f04539b084ee778e9ed8a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 06:22:27.233613 18730 tablet_replica.cc:439] T b674d97a62f042deb30e4ec72a7572d8 P 588e6c5989f04539b084ee778e9ed8a7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } } }
I20260807 06:22:27.233736 18730 tablet_replica.cc:442] T b674d97a62f042deb30e4ec72a7572d8 P 588e6c5989f04539b084ee778e9ed8a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:22:27.235368 18708 ts_tablet_manager.cc:1428] T b674d97a62f042deb30e4ec72a7572d8 P f0654852d6404e6488ebce8e50bf9943: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20260807 06:22:27.297505 18719 raft_consensus.cc:482] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:22:27.297674 18719 raft_consensus.cc:504] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } }
I20260807 06:22:27.298868 18719 leader_election.cc:288] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0654852d6404e6488ebce8e50bf9943 (127.15.254.1:36583), 588e6c5989f04539b084ee778e9ed8a7 (127.15.254.3:45095)
I20260807 06:22:27.303985 18770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:27.299118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:36583 (local address 127.15.254.2:54897)
0807 06:22:27.299531 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:27.299544 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:27.299694 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:27.300014 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:27.300016 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:27.300036 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:27.300242 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:27.300250 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.301237 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.301240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:27.302511 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:27.302522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.303386 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.303390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:27.303595 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:27.303637 (+ 42us) client_negotiation.cc:769] Sending connection context
0807 06:22:27.303714 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 06:22:27.303794 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":24,"thread_start_us":174,"threads_started":1}
I20260807 06:22:27.303992 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:27.299264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:54897 (local address 127.15.254.1:36583)
0807 06:22:27.299438 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:27.299445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:27.299640 (+ 195us) server_negotiation.cc:407] Connection header received
0807 06:22:27.299829 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:27.299835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:27.299895 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:27.299980 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:27.300373 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.301106 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.302634 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.303251 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.303353 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:27.303398 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:27.303751 (+ 353us) server_negotiation.cc:299] Negotiation successful
0807 06:22:27.303840 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 06:22:27.304284 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:27.299749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:36659 (local address 127.15.254.3:45095)
0807 06:22:27.299917 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:27.299928 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:27.300218 (+ 290us) server_negotiation.cc:407] Connection header received
0807 06:22:27.300287 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:27.300293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:27.300353 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:27.300424 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:27.300904 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.301646 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.302967 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.303552 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.303681 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:27.303749 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:27.304053 (+ 304us) server_negotiation.cc:299] Negotiation successful
0807 06:22:27.304150 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 06:22:27.304570 18772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:27.299624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:45095 (local address 127.15.254.2:36659)
0807 06:22:27.300051 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:27.300066 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:27.300216 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:27.300466 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:27.300469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:27.300511 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:27.300691 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:27.300699 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.301772 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.301775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:27.302791 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:27.302803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.303730 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.303735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:27.303862 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:27.303886 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 06:22:27.303976 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 06:22:27.304389 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":86,"thread_start_us":112,"threads_started":1}
I20260807 06:22:27.304553 18206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b674d97a62f042deb30e4ec72a7572d8" candidate_uuid: "3a95e762028341a59c1f85991a16607a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0654852d6404e6488ebce8e50bf9943" is_pre_election: true
I20260807 06:22:27.304702 18206 raft_consensus.cc:2454] T b674d97a62f042deb30e4ec72a7572d8 P f0654852d6404e6488ebce8e50bf9943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a95e762028341a59c1f85991a16607a in term 0.
I20260807 06:22:27.305155 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b674d97a62f042deb30e4ec72a7572d8" candidate_uuid: "3a95e762028341a59c1f85991a16607a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588e6c5989f04539b084ee778e9ed8a7" is_pre_election: true
I20260807 06:22:27.305212 18303 leader_election.cc:302] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a95e762028341a59c1f85991a16607a, f0654852d6404e6488ebce8e50bf9943; no voters:
I20260807 06:22:27.305413 18608 raft_consensus.cc:2454] T b674d97a62f042deb30e4ec72a7572d8 P 588e6c5989f04539b084ee778e9ed8a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a95e762028341a59c1f85991a16607a in term 0.
I20260807 06:22:27.305533 18719 raft_consensus.cc:2781] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:22:27.305624 18719 raft_consensus.cc:482] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:22:27.305683 18719 raft_consensus.cc:3037] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:27.306874 18719 raft_consensus.cc:504] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } }
I20260807 06:22:27.307262 18719 leader_election.cc:288] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a [CANDIDATE]: Term 1 election: Requested vote from peers f0654852d6404e6488ebce8e50bf9943 (127.15.254.1:36583), 588e6c5989f04539b084ee778e9ed8a7 (127.15.254.3:45095)
I20260807 06:22:27.307715 18206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b674d97a62f042deb30e4ec72a7572d8" candidate_uuid: "3a95e762028341a59c1f85991a16607a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0654852d6404e6488ebce8e50bf9943"
I20260807 06:22:27.307734 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b674d97a62f042deb30e4ec72a7572d8" candidate_uuid: "3a95e762028341a59c1f85991a16607a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588e6c5989f04539b084ee778e9ed8a7"
I20260807 06:22:27.307845 18206 raft_consensus.cc:3037] T b674d97a62f042deb30e4ec72a7572d8 P f0654852d6404e6488ebce8e50bf9943 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:27.307844 18608 raft_consensus.cc:3037] T b674d97a62f042deb30e4ec72a7572d8 P 588e6c5989f04539b084ee778e9ed8a7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:27.309046 18206 raft_consensus.cc:2454] T b674d97a62f042deb30e4ec72a7572d8 P f0654852d6404e6488ebce8e50bf9943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a95e762028341a59c1f85991a16607a in term 1.
I20260807 06:22:27.309046 18608 raft_consensus.cc:2454] T b674d97a62f042deb30e4ec72a7572d8 P 588e6c5989f04539b084ee778e9ed8a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a95e762028341a59c1f85991a16607a in term 1.
I20260807 06:22:27.309406 18303 leader_election.cc:302] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a95e762028341a59c1f85991a16607a, f0654852d6404e6488ebce8e50bf9943; no voters:
I20260807 06:22:27.309613 18719 raft_consensus.cc:2781] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:27.309830 18719 raft_consensus.cc:686] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a [term 1 LEADER]: Becoming Leader. State: Replica: 3a95e762028341a59c1f85991a16607a, State: Running, Role: LEADER
I20260807 06:22:27.310137 18719 consensus_queue.cc:237] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } }
I20260807 06:22:27.312245 18773 tablet_replica.cc:439] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a: TxnStatusTablet state changed. Reason: New leader 3a95e762028341a59c1f85991a16607a. Latest consensus state: current_term: 1 leader_uuid: "3a95e762028341a59c1f85991a16607a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } } }
I20260807 06:22:27.312400 18773 tablet_replica.cc:442] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a: This TxnStatusTablet replica's current role is: LEADER
I20260807 06:22:27.312830 18775 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 06:22:27.313145 18775 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 06:22:27.313616 17982 catalog_manager.cc:5495] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a95e762028341a59c1f85991a16607a (127.15.254.2). New cstate: current_term: 1 leader_uuid: "3a95e762028341a59c1f85991a16607a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } health_report { overall_health: UNKNOWN } } }
I20260807 06:22:27.338340 18758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:27.333962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:40227 (local address 127.0.0.1:40242)
0807 06:22:27.334226 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:27.334243 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:27.334386 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:27.334740 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:27.334743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:27.334751 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 06:22:27.334866 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:27.334872 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.335655 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.335659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:27.336294 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:27.336302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.336909 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.336912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:27.337009 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:27.337076 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 06:22:27.338021 (+ 945us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 06:22:27.338041 (+ 20us) client_negotiation.cc:769] Sending connection context
0807 06:22:27.338123 (+ 82us) client_negotiation.cc:240] Negotiation successful
0807 06:22:27.338215 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260807 06:22:27.338393 18702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:27.334053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40242 (local address 127.15.254.2:40227)
0807 06:22:27.334185 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:27.334190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:27.334315 (+ 125us) server_negotiation.cc:407] Connection header received
0807 06:22:27.334539 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:27.334545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:27.334613 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:27.334724 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:22:27.334985 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.335513 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.336441 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.336789 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.336907 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:27.337245 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:22:27.337919 (+ 674us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:22:27.338050 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:27.338134 (+ 84us) server_negotiation.cc:299] Negotiation successful
0807 06:22:27.338191 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 06:22:27.341711 18206 raft_consensus.cc:1261] T b674d97a62f042deb30e4ec72a7572d8 P f0654852d6404e6488ebce8e50bf9943 [term 1 FOLLOWER]: Refusing update from remote peer 3a95e762028341a59c1f85991a16607a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:22:27.341908 18608 raft_consensus.cc:1261] T b674d97a62f042deb30e4ec72a7572d8 P 588e6c5989f04539b084ee778e9ed8a7 [term 1 FOLLOWER]: Refusing update from remote peer 3a95e762028341a59c1f85991a16607a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:22:27.342298 18773 consensus_queue.cc:1035] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:22:27.342528 18719 consensus_queue.cc:1035] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a [LEADER]: Connected to new peer: Peer: permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:22:27.344149 18717 tablet_replica.cc:439] T b674d97a62f042deb30e4ec72a7572d8 P f0654852d6404e6488ebce8e50bf9943: TxnStatusTablet state changed. Reason: New leader 3a95e762028341a59c1f85991a16607a. Latest consensus state: current_term: 1 leader_uuid: "3a95e762028341a59c1f85991a16607a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } } }
I20260807 06:22:27.344269 18717 tablet_replica.cc:442] T b674d97a62f042deb30e4ec72a7572d8 P f0654852d6404e6488ebce8e50bf9943: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:22:27.344434 18730 tablet_replica.cc:439] T b674d97a62f042deb30e4ec72a7572d8 P 588e6c5989f04539b084ee778e9ed8a7: TxnStatusTablet state changed. Reason: New leader 3a95e762028341a59c1f85991a16607a. Latest consensus state: current_term: 1 leader_uuid: "3a95e762028341a59c1f85991a16607a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } } }
I20260807 06:22:27.344568 18730 tablet_replica.cc:442] T b674d97a62f042deb30e4ec72a7572d8 P 588e6c5989f04539b084ee778e9ed8a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:22:27.346330 18773 tablet_replica.cc:439] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a95e762028341a59c1f85991a16607a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } } }
I20260807 06:22:27.346457 18773 tablet_replica.cc:442] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a: This TxnStatusTablet replica's current role is: LEADER
I20260807 06:22:27.346890 18726 mvcc.cc:205] Tried to move back new op lower bound from 7315799029105672192 to 7315799028985270272. Current Snapshot: MvccSnapshot[applied={T|T < 7315799029105672192}]
I20260807 06:22:27.349021 18717 tablet_replica.cc:439] T b674d97a62f042deb30e4ec72a7572d8 P f0654852d6404e6488ebce8e50bf9943: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a95e762028341a59c1f85991a16607a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } } }
I20260807 06:22:27.349130 18717 tablet_replica.cc:442] T b674d97a62f042deb30e4ec72a7572d8 P f0654852d6404e6488ebce8e50bf9943: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:22:27.357853 18773 tablet_replica.cc:439] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a95e762028341a59c1f85991a16607a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } } }
I20260807 06:22:27.358004 18773 tablet_replica.cc:442] T b674d97a62f042deb30e4ec72a7572d8 P 3a95e762028341a59c1f85991a16607a: This TxnStatusTablet replica's current role is: LEADER
I20260807 06:22:27.359217 18725 mvcc.cc:205] Tried to move back new op lower bound from 7315799029105672192 to 7315799028985270272. Current Snapshot: MvccSnapshot[applied={T|T < 7315799029105672192}]
I20260807 06:22:27.359423 18730 tablet_replica.cc:439] T b674d97a62f042deb30e4ec72a7572d8 P 588e6c5989f04539b084ee778e9ed8a7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a95e762028341a59c1f85991a16607a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 } } peers { permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 } } peers { permanent_uuid: "588e6c5989f04539b084ee778e9ed8a7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 45095 } } }
I20260807 06:22:27.359544 18730 tablet_replica.cc:442] T b674d97a62f042deb30e4ec72a7572d8 P 588e6c5989f04539b084ee778e9ed8a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:22:27.364637 18702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:27.360784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:47991 (local address 127.15.254.2:40227)
0807 06:22:27.360925 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:27.360930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:27.361069 (+ 139us) server_negotiation.cc:407] Connection header received
0807 06:22:27.361286 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:27.361289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:27.361354 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:27.361421 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:22:27.361695 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.362248 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.363549 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.363765 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.363841 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:27.364166 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:22:27.364232 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:22:27.364286 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:27.364408 (+ 122us) server_negotiation.cc:299] Negotiation successful
0807 06:22:27.364453 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 06:22:27.364645 18789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:27.360159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:40227 (local address 127.15.254.3:47991)
0807 06:22:27.360982 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:27.360999 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:27.361150 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:27.361456 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:27.361459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:27.361469 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 06:22:27.361583 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:27.361590 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.362351 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.362355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:27.363420 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:27.363435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.363870 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.363873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:27.363974 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:27.364030 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 06:22:27.364312 (+ 282us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 06:22:27.364319 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 06:22:27.364386 (+ 67us) client_negotiation.cc:240] Negotiation successful
0807 06:22:27.364474 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":86,"threads_started":1}
I20260807 06:22:27.371151 18402 raft_consensus.cc:1261] T 24845fb791d74e909bb6a7b1596e9fbe P 3a95e762028341a59c1f85991a16607a [term 1 FOLLOWER]: Refusing update from remote peer 588e6c5989f04539b084ee778e9ed8a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:22:27.371284 18206 raft_consensus.cc:1261] T 24845fb791d74e909bb6a7b1596e9fbe P f0654852d6404e6488ebce8e50bf9943 [term 1 FOLLOWER]: Refusing update from remote peer 588e6c5989f04539b084ee778e9ed8a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:22:27.371574 18721 consensus_queue.cc:1035] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a95e762028341a59c1f85991a16607a" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 40227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:22:27.371722 18730 consensus_queue.cc:1035] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0654852d6404e6488ebce8e50bf9943" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:22:27.397078 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:27.392897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:36381 (local address 127.15.254.3:45095)
0807 06:22:27.393057 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:27.393063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:27.393433 (+ 370us) server_negotiation.cc:407] Connection header received
0807 06:22:27.393564 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:27.393567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:27.393618 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:27.393695 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:22:27.394031 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.394590 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.395952 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.396201 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.396284 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:27.396647 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:22:27.396719 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:22:27.396777 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:27.396899 (+ 122us) server_negotiation.cc:299] Negotiation successful
0807 06:22:27.396948 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 06:22:27.397164 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:27.392840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:45095 (local address 127.15.254.2:36381)
0807 06:22:27.393316 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:27.393337 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:27.393473 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:27.393724 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:27.393727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:27.393735 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 06:22:27.393887 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:27.393894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.394712 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.394716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:27.395793 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:27.395805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.396313 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.396317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:27.396435 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:27.396516 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 06:22:27.396806 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 06:22:27.396814 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 06:22:27.396891 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 06:22:27.396984 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":170,"threads_started":1}
I20260807 06:22:27.422987 17981 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:22:27.423182 17981 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:22:27.425387 17981 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 099cd300fbd84bcfbcd9554503e7a55f: Sending DeleteTablet for 3 replicas of tablet 24845fb791d74e909bb6a7b1596e9fbe
I20260807 06:22:27.425951 18180 tablet_service.cc:1514] Processing DeleteTablet for tablet 24845fb791d74e909bb6a7b1596e9fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:27 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:60116
I20260807 06:22:27.426156 18743 tablet_replica.cc:330] T 24845fb791d74e909bb6a7b1596e9fbe P f0654852d6404e6488ebce8e50bf9943: stopping tablet replica
I20260807 06:22:27.426290 18743 raft_consensus.cc:2229] T 24845fb791d74e909bb6a7b1596e9fbe P f0654852d6404e6488ebce8e50bf9943 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:22:27.426357 18582 tablet_service.cc:1514] Processing DeleteTablet for tablet 24845fb791d74e909bb6a7b1596e9fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:27 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:37846
I20260807 06:22:27.426414 18743 raft_consensus.cc:2258] T 24845fb791d74e909bb6a7b1596e9fbe P f0654852d6404e6488ebce8e50bf9943 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:27.426532 18742 tablet_replica.cc:330] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7: stopping tablet replica
I20260807 06:22:27.426662 18742 raft_consensus.cc:2229] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:22:27.426872 18742 raft_consensus.cc:2258] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:27.427261 18743 ts_tablet_manager.cc:1905] T 24845fb791d74e909bb6a7b1596e9fbe P f0654852d6404e6488ebce8e50bf9943: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:22:27.427239 18376 tablet_service.cc:1514] Processing DeleteTablet for tablet 24845fb791d74e909bb6a7b1596e9fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:27 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:40234
I20260807 06:22:27.427423 18741 tablet_replica.cc:330] T 24845fb791d74e909bb6a7b1596e9fbe P 3a95e762028341a59c1f85991a16607a: stopping tablet replica
I20260807 06:22:27.427533 18741 raft_consensus.cc:2229] T 24845fb791d74e909bb6a7b1596e9fbe P 3a95e762028341a59c1f85991a16607a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:22:27.427646 18741 raft_consensus.cc:2258] T 24845fb791d74e909bb6a7b1596e9fbe P 3a95e762028341a59c1f85991a16607a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:27.428290 18742 ts_tablet_manager.cc:1905] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:22:27.430513 18743 ts_tablet_manager.cc:1918] T 24845fb791d74e909bb6a7b1596e9fbe P f0654852d6404e6488ebce8e50bf9943: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:22:27.430625 18743 log.cc:1196] T 24845fb791d74e909bb6a7b1596e9fbe P f0654852d6404e6488ebce8e50bf9943: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-0/wal/wals/24845fb791d74e909bb6a7b1596e9fbe
I20260807 06:22:27.430889 18743 ts_tablet_manager.cc:1939] T 24845fb791d74e909bb6a7b1596e9fbe P f0654852d6404e6488ebce8e50bf9943: Deleting consensus metadata
I20260807 06:22:27.431427 18742 ts_tablet_manager.cc:1918] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:22:27.431516 18742 log.cc:1196] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/24845fb791d74e909bb6a7b1596e9fbe
I20260807 06:22:27.431684 17958 catalog_manager.cc:4841] TS f0654852d6404e6488ebce8e50bf9943 (127.15.254.1:36583): tablet 24845fb791d74e909bb6a7b1596e9fbe (table test-table [id=4e46f5fe4a9e48648ac3bb4ad4afc7e1]) successfully deleted
I20260807 06:22:27.432080 18742 ts_tablet_manager.cc:1939] T 24845fb791d74e909bb6a7b1596e9fbe P 588e6c5989f04539b084ee778e9ed8a7: Deleting consensus metadata
I20260807 06:22:27.432874 17958 catalog_manager.cc:4841] TS 588e6c5989f04539b084ee778e9ed8a7 (127.15.254.3:45095): tablet 24845fb791d74e909bb6a7b1596e9fbe (table test-table [id=4e46f5fe4a9e48648ac3bb4ad4afc7e1]) successfully deleted
I20260807 06:22:27.433382 18741 ts_tablet_manager.cc:1905] T 24845fb791d74e909bb6a7b1596e9fbe P 3a95e762028341a59c1f85991a16607a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:22:27.436425 18741 ts_tablet_manager.cc:1918] T 24845fb791d74e909bb6a7b1596e9fbe P 3a95e762028341a59c1f85991a16607a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:22:27.436538 18741 log.cc:1196] T 24845fb791d74e909bb6a7b1596e9fbe P 3a95e762028341a59c1f85991a16607a: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/24845fb791d74e909bb6a7b1596e9fbe
I20260807 06:22:27.436887 18741 ts_tablet_manager.cc:1939] T 24845fb791d74e909bb6a7b1596e9fbe P 3a95e762028341a59c1f85991a16607a: Deleting consensus metadata
I20260807 06:22:27.437762 17958 catalog_manager.cc:4841] TS 3a95e762028341a59c1f85991a16607a (127.15.254.2:40227): tablet 24845fb791d74e909bb6a7b1596e9fbe (table test-table [id=4e46f5fe4a9e48648ac3bb4ad4afc7e1]) successfully deleted
Aug 07 06:22:27 dist-test-slave-44d2 krb5kdc[17916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083746, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.254.1@KRBTEST.COM
I20260807 06:22:27.443302 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:27.432040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60130 (local address 127.15.254.1:36583)
0807 06:22:27.432157 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:27.432162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:27.432468 (+ 306us) server_negotiation.cc:407] Connection header received
0807 06:22:27.432621 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:27.432624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:27.432683 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:27.432768 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:27.433763 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.434482 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.435705 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.436121 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.436209 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:27.439607 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:27.439619 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:27.439636 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:27.439670 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:27.441378 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:27.441877 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:27.441884 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:27.441885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:27.441968 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:27.442296 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:27.442300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:27.442302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:27.442561 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:27.442786 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:27.442989 (+ 203us) server_negotiation.cc:299] Negotiation successful
0807 06:22:27.443133 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260807 06:22:27.443794 18258 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60130
I20260807 06:22:27.456351 18759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:27.445690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42992 (local address 127.15.254.62:39757)
0807 06:22:27.445810 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:27.445815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:27.447491 (+ 1676us) server_negotiation.cc:407] Connection header received
0807 06:22:27.447694 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:27.447697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:27.447747 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:27.447794 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:27.448661 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.449497 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.450644 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:27.450997 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:27.451063 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:27.452926 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:27.452930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:27.452933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:27.452969 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:27.454428 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:27.454884 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:27.454888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:27.454890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:27.455031 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:27.455337 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:27.455341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:27.455342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:27.455587 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:27.455735 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:27.456001 (+ 266us) server_negotiation.cc:299] Negotiation successful
0807 06:22:27.456159 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260807 06:22:27.456861 17981 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42992
I20260807 06:22:27.458091 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 18081
I20260807 06:22:27.470078 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 18278
I20260807 06:22:27.483251 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 18482
I20260807 06:22:27.500123 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 17933
2026-08-07T06:22:27Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3295 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:22:27 dist-test-slave-44d2 krb5kdc[18808](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:22:27 dist-test-slave-44d2 krb5kdc[18808](info): set up 2 sockets
Aug 07 06:22:27 dist-test-slave-44d2 krb5kdc[18808](info): commencing operation
krb5kdc: starting...
W20260807 06:22:28.728772 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.202s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:22:28 dist-test-slave-44d2 krb5kdc[18808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:22:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:22:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:22:28.868256 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:44813
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:46841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:44813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:28.986476 18829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:28.986766 18829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:28.986809 18829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:28.991007 18829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:22:28.991087 18829 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 06:22:28.991106 18829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:28.991125 18829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:28.991142 18829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:22:28.991158 18829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:22:28.996929 18829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:46841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:44813
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:44813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:28.998082 18829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:28.999179 18829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:29.008569 18845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:29.008595 18848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:29.008661 18829 server_base.cc:1034] running on GCE node
W20260807 06:22:29.008595 18841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:29.009423 18829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:29.010632 18829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:29.011824 18829 hybrid_clock.cc:648] HybridClock initialized: now 1786083749011790 us; error 47 us; skew 500 ppm
Aug 07 06:22:29 dist-test-slave-44d2 krb5kdc[18808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083749, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:29.015223 18829 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:22:29.016410 18829 webserver.cc:460] Webserver started at http://127.15.254.62:38837/ using document root <none> and password file <none>
I20260807 06:22:29.017135 18829 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:29.017243 18829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:29.017503 18829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:29.019860 18829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "09fb44b5c342445c84c0a88d46b1d8ce"
format_stamp: "Formatted at 2026-08-07 06:22:29 on dist-test-slave-44d2"
server_key: "6aa4a07eeaed6beb68d78db6d30d4444"
server_key_iv: "d27ad534982a5f2591e9aece0cee2bdd"
server_key_version: "encryptionkey@0"
I20260807 06:22:29.020565 18829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "09fb44b5c342445c84c0a88d46b1d8ce"
format_stamp: "Formatted at 2026-08-07 06:22:29 on dist-test-slave-44d2"
server_key: "6aa4a07eeaed6beb68d78db6d30d4444"
server_key_iv: "d27ad534982a5f2591e9aece0cee2bdd"
server_key_version: "encryptionkey@0"
I20260807 06:22:29.024736 18829 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 06:22:29.027484 18856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:29.028959 18829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:22:29.029130 18829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "09fb44b5c342445c84c0a88d46b1d8ce"
format_stamp: "Formatted at 2026-08-07 06:22:29 on dist-test-slave-44d2"
server_key: "6aa4a07eeaed6beb68d78db6d30d4444"
server_key_iv: "d27ad534982a5f2591e9aece0cee2bdd"
server_key_version: "encryptionkey@0"
I20260807 06:22:29.029237 18829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:29.060767 18829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:29.064090 18829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:29.064309 18829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:29.072974 18829 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:44813
I20260807 06:22:29.072989 18946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:44813 every 8 connection(s)
I20260807 06:22:29.073930 18829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:22:29.076608 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 18829
I20260807 06:22:29.076725 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:22:29.076865 18947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:29.077035 16376 external_mini_cluster.cc:1428] Setting key 408e8a54c0c741c142fda79cf9276e6e
I20260807 06:22:29.083353 18947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce: Bootstrap starting.
I20260807 06:22:29.085850 18947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce: Neither blocks nor log segments found. Creating new log.
Aug 07 06:22:29 dist-test-slave-44d2 krb5kdc[18808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:29.086604 18947 log.cc:824] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:29.088838 18947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce: No bootstrap required, opened a new log
I20260807 06:22:29.091974 18947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09fb44b5c342445c84c0a88d46b1d8ce" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 44813 } }
I20260807 06:22:29.092128 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:29.078498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59970 (local address 127.15.254.62:44813)
0807 06:22:29.078963 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:29.078978 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:29.079014 (+ 36us) server_negotiation.cc:407] Connection header received
0807 06:22:29.079695 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:29.079713 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:29.080040 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:29.080417 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:29.081258 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.082269 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.083145 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.083686 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.083853 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:29.086501 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:29.086506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:29.086519 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:29.086548 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:29.089166 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:29.089680 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:29.089686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:29.089691 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:29.089818 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:29.090132 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:29.090136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:29.090137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:29.090522 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:29.090747 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:29.091052 (+ 305us) server_negotiation.cc:299] Negotiation successful
0807 06:22:29.091369 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":150,"threads_started":1}
I20260807 06:22:29.092201 18947 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:29.092267 18947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09fb44b5c342445c84c0a88d46b1d8ce, State: Initialized, Role: FOLLOWER
I20260807 06:22:29.092762 18947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09fb44b5c342445c84c0a88d46b1d8ce" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 44813 } }
I20260807 06:22:29.092916 18947 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:22:29.092998 18947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:22:29.093117 18947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:29.093990 18947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09fb44b5c342445c84c0a88d46b1d8ce" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 44813 } }
I20260807 06:22:29.094386 18947 leader_election.cc:302] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09fb44b5c342445c84c0a88d46b1d8ce; no voters:
I20260807 06:22:29.094723 18947 leader_election.cc:288] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:22:29.094848 18958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:29.095095 18958 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce [term 1 LEADER]: Becoming Leader. State: Replica: 09fb44b5c342445c84c0a88d46b1d8ce, State: Running, Role: LEADER
I20260807 06:22:29.095399 18958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09fb44b5c342445c84c0a88d46b1d8ce" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 44813 } }
I20260807 06:22:29.095938 18947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:22:29.096560 18958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09fb44b5c342445c84c0a88d46b1d8ce. Latest consensus state: current_term: 1 leader_uuid: "09fb44b5c342445c84c0a88d46b1d8ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09fb44b5c342445c84c0a88d46b1d8ce" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 44813 } } }
I20260807 06:22:29.096724 18958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:29.097066 18961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09fb44b5c342445c84c0a88d46b1d8ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09fb44b5c342445c84c0a88d46b1d8ce" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 44813 } } }
I20260807 06:22:29.097182 18961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:29.097184 18967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:22:29.100724 18967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:22:29.106040 18967 catalog_manager.cc:1347] Generated new cluster ID: c02b70d76e9445c8b92c7fb8043d627b
I20260807 06:22:29.106108 18967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:22:29.118019 18967 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:22:29.119234 18967 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:22:29.134059 18967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce: Generated new TSK 0
I20260807 06:22:29.134856 18967 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
I20260807 06:22:29.179836 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:0
--local_ip_for_outbound_sockets=127.15.254.1
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:44813
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:46841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:29.307343 18990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:29.307626 18990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:29.307703 18990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:29.312361 18990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:29.312426 18990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:29.312469 18990 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 06:22:29.312534 18990 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 06:22:29.312631 18990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:22:29.318085 18990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:46841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.254.62:44813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:29.319201 18990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:29.320261 18990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:29.330252 18998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:29.330358 19001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:29.330260 19006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:29.330596 18990 server_base.cc:1034] running on GCE node
I20260807 06:22:29.331009 18990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:29.331689 18990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:29.332873 18990 hybrid_clock.cc:648] HybridClock initialized: now 1786083749332845 us; error 50 us; skew 500 ppm
Aug 07 06:22:29 dist-test-slave-44d2 krb5kdc[18808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083749, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:29.346766 18990 init.cc:377] Logged in from keytab as kudu/127.15.254.1@KRBTEST.COM (short username kudu)
I20260807 06:22:29.347882 18990 webserver.cc:460] Webserver started at http://127.15.254.1:44623/ using document root <none> and password file <none>
I20260807 06:22:29.348472 18990 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:29.348608 18990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:29.348802 18990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:29.350932 18990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-0/data/instance:
uuid: "f4b98f4bd4d64a549306c2abd1f0851a"
format_stamp: "Formatted at 2026-08-07 06:22:29 on dist-test-slave-44d2"
server_key: "576698b805bdfb1bd90cc268465a4aa4"
server_key_iv: "2ee20a1cefdd09e6d7818398b5628e2f"
server_key_version: "encryptionkey@0"
I20260807 06:22:29.351483 18990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance:
uuid: "f4b98f4bd4d64a549306c2abd1f0851a"
format_stamp: "Formatted at 2026-08-07 06:22:29 on dist-test-slave-44d2"
server_key: "576698b805bdfb1bd90cc268465a4aa4"
server_key_iv: "2ee20a1cefdd09e6d7818398b5628e2f"
server_key_version: "encryptionkey@0"
I20260807 06:22:29.355418 18990 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260807 06:22:29.358011 19023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:29.359292 18990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 06:22:29.359429 18990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "f4b98f4bd4d64a549306c2abd1f0851a"
format_stamp: "Formatted at 2026-08-07 06:22:29 on dist-test-slave-44d2"
server_key: "576698b805bdfb1bd90cc268465a4aa4"
server_key_iv: "2ee20a1cefdd09e6d7818398b5628e2f"
server_key_version: "encryptionkey@0"
I20260807 06:22:29.359539 18990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:29.377556 18990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:29.381111 18990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:29.381330 18990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:29.382987 18990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:29.383041 18990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:29.383090 18990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:29.383143 18990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 06:22:29 dist-test-slave-44d2 krb5kdc[18808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083749, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:29.394987 18990 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:33979
I20260807 06:22:29.395424 19202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:33979 every 8 connection(s)
I20260807 06:22:29.396029 18990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:22:29.398377 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:29.383990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:42489 (local address 127.15.254.62:44813)
0807 06:22:29.384135 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:29.384139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:29.385274 (+ 1135us) server_negotiation.cc:407] Connection header received
0807 06:22:29.386387 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:29.386392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:29.386461 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:29.386584 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:29.387961 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.388640 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.389947 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.390187 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.390303 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:29.394007 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:29.394011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:29.394014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:29.394051 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:29.395981 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:29.396658 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:29.396662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:29.396664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:29.396805 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:29.397288 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:29.397291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:29.397293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:29.397483 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:29.397566 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:29.398149 (+ 583us) server_negotiation.cc:299] Negotiation successful
0807 06:22:29.398256 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 06:22:29.398710 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 18990
I20260807 06:22:29.398835 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance
I20260807 06:22:29.399137 16376 external_mini_cluster.cc:1428] Setting key 7d4cb2922f97d131f326e8426c70608e
I20260807 06:22:29.399397 19038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:29.384558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:44813 (local address 127.15.254.1:42489)
0807 06:22:29.385118 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:29.385162 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:29.386084 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:29.386770 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:29.386777 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:29.387297 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:29.387783 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:29.387824 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.388775 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.388782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:29.389811 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:29.389821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.390344 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.390348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:29.390557 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:29.391187 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:29.391203 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:29.393804 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:29.396141 (+ 2337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:29.396147 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:29.396161 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:29.396551 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:29.396951 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:29.396954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:29.396956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:29.397134 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:29.397575 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:29.397580 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:29.397789 (+ 209us) client_negotiation.cc:769] Sending connection context
0807 06:22:29.398068 (+ 279us) client_negotiation.cc:240] Negotiation successful
0807 06:22:29.398321 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":161,"threads_started":1}
I20260807 06:22:29.400858 19203 heartbeater.cc:347] Connected to a master server at 127.15.254.62:44813
I20260807 06:22:29.401016 19203 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:29.401603 19203 heartbeater.cc:511] Master 127.15.254.62:44813 requested a full tablet report, sending...
I20260807 06:22:29.403280 18882 ts_manager.cc:194] Registered new tserver with Master: f4b98f4bd4d64a549306c2abd1f0851a (127.15.254.1:33979)
I20260807 06:22:29.404438 18882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.1@KRBTEST.COM'} at 127.15.254.1:42489
WARNING: no policy specified for kudu/127.15.254.2@KRBTEST.COM; defaulting to no policy
I20260807 06:22:29.413136 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:29.405140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:45035 (local address 127.15.254.62:44813)
0807 06:22:29.405271 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:29.405275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:29.405555 (+ 280us) server_negotiation.cc:407] Connection header received
0807 06:22:29.405725 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:29.405728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:29.405775 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:29.405860 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:29.406390 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.407034 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.408431 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.408694 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.408777 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:29.410113 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:29.410116 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:29.410117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:29.410140 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:29.411578 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:29.411984 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:29.411986 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:29.411988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:29.412061 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:29.412361 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:29.412363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:29.412364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:29.412610 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:29.412692 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:29.412920 (+ 228us) server_negotiation.cc:299] Negotiation successful
0807 06:22:29.413011 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 06:22:29.413154 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:29.404771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:44813 (local address 127.15.254.1:45035)
0807 06:22:29.405455 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:29.405469 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:29.405618 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:29.405877 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:29.405880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:29.406119 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:29.406269 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:29.406275 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.407210 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.407217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:29.408286 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:29.408301 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.408815 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.408825 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:29.408936 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:29.409470 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:29.409489 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:29.409961 (+ 472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:29.411687 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:29.411690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:29.411691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:29.411895 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:29.412162 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:29.412169 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:29.412171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:29.412276 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:29.412699 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:29.412701 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:29.412781 (+ 80us) client_negotiation.cc:769] Sending connection context
0807 06:22:29.412887 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 06:22:29.413008 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":2,"thread_start_us":134,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
I20260807 06:22:29.432431 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:0
--local_ip_for_outbound_sockets=127.15.254.2
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:44813
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:46841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:29.550892 19216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:29.551206 19216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:29.551290 19216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:29.555436 19216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:29.555531 19216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:29.555606 19216 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 06:22:29.555649 19216 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 06:22:29.555720 19216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:22:29.561087 19216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:46841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.254.62:44813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:29.562572 19216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:29.563992 19216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:29.579082 19226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:29.579142 19216 server_base.cc:1034] running on GCE node
W20260807 06:22:29.579417 19221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:29.580169 19223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:29.580932 19216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:29.582024 19216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:29.583245 19216 hybrid_clock.cc:648] HybridClock initialized: now 1786083749583200 us; error 47 us; skew 500 ppm
Aug 07 06:22:29 dist-test-slave-44d2 krb5kdc[18808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083749, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:29.587283 19216 init.cc:377] Logged in from keytab as kudu/127.15.254.2@KRBTEST.COM (short username kudu)
I20260807 06:22:29.589587 19216 webserver.cc:460] Webserver started at http://127.15.254.2:33231/ using document root <none> and password file <none>
I20260807 06:22:29.590355 19216 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:29.590463 19216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:29.590772 19216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:29.593786 19216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-1/data/instance:
uuid: "c2c6d77d1d7c4b1888a2d1385067cedf"
format_stamp: "Formatted at 2026-08-07 06:22:29 on dist-test-slave-44d2"
server_key: "6b3ee44dbd1f042c9b57a1785c61d438"
server_key_iv: "6d67a38c3516175df88be6125c8cd844"
server_key_version: "encryptionkey@0"
I20260807 06:22:29.594480 19216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance:
uuid: "c2c6d77d1d7c4b1888a2d1385067cedf"
format_stamp: "Formatted at 2026-08-07 06:22:29 on dist-test-slave-44d2"
server_key: "6b3ee44dbd1f042c9b57a1785c61d438"
server_key_iv: "6d67a38c3516175df88be6125c8cd844"
server_key_version: "encryptionkey@0"
I20260807 06:22:29.600265 19216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 06:22:29.603843 19237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:29.605508 19216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:22:29.605676 19216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "c2c6d77d1d7c4b1888a2d1385067cedf"
format_stamp: "Formatted at 2026-08-07 06:22:29 on dist-test-slave-44d2"
server_key: "6b3ee44dbd1f042c9b57a1785c61d438"
server_key_iv: "6d67a38c3516175df88be6125c8cd844"
server_key_version: "encryptionkey@0"
I20260807 06:22:29.605832 19216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:29.625196 19216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:29.628571 19216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:29.628729 19216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:29.630293 19216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:29.630345 19216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:29.630407 19216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:29.630438 19216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 06:22:29 dist-test-slave-44d2 krb5kdc[18808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083749, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:29.641835 19216 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:39045
I20260807 06:22:29.641933 19405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:39045 every 8 connection(s)
I20260807 06:22:29.643002 19216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:22:29.645960 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:29.631166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:52729 (local address 127.15.254.62:44813)
0807 06:22:29.631293 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:29.631297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:29.632441 (+ 1144us) server_negotiation.cc:407] Connection header received
0807 06:22:29.633858 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:29.633864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:29.633953 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:29.634045 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:29.636070 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.636956 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.638809 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.639018 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.639103 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:29.642304 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:29.642308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:29.642311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:29.642341 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:29.644031 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:29.644574 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:29.644578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:29.644580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:29.644666 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:29.644973 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:29.644975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:29.644976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:29.645162 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:29.645238 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:29.645731 (+ 493us) server_negotiation.cc:299] Negotiation successful
0807 06:22:29.645834 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 06:22:29.646808 19252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:29.631740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:44813 (local address 127.15.254.2:52729)
0807 06:22:29.632273 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:29.632349 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:29.633605 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:29.634678 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:29.634693 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:29.635155 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:29.635899 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:29.635925 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.637114 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.637118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:29.638670 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:29.638682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.639187 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.639193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:29.639425 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:29.640175 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:29.640226 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:29.642126 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:29.644168 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:29.644172 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:29.644183 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:29.644429 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:29.644750 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:29.644752 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:29.644754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:29.644887 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:29.645249 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:29.645254 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:29.645507 (+ 253us) client_negotiation.cc:769] Sending connection context
0807 06:22:29.645670 (+ 163us) client_negotiation.cc:240] Negotiation successful
0807 06:22:29.645906 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":138,"threads_started":1}
I20260807 06:22:29.648121 19407 heartbeater.cc:347] Connected to a master server at 127.15.254.62:44813
I20260807 06:22:29.648262 19407 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:29.648824 19407 heartbeater.cc:511] Master 127.15.254.62:44813 requested a full tablet report, sending...
I20260807 06:22:29.649926 18879 ts_manager.cc:194] Registered new tserver with Master: c2c6d77d1d7c4b1888a2d1385067cedf (127.15.254.2:39045)
I20260807 06:22:29.650628 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 19216
I20260807 06:22:29.650756 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance
I20260807 06:22:29.650720 18879 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.2@KRBTEST.COM'} at 127.15.254.2:52729
I20260807 06:22:29.651091 16376 external_mini_cluster.cc:1428] Setting key 4114ce6797352e06b17d8b52764bfe12
I20260807 06:22:29.658754 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:29.650568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:48721 (local address 127.15.254.62:44813)
0807 06:22:29.650737 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:29.650744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:29.650991 (+ 247us) server_negotiation.cc:407] Connection header received
0807 06:22:29.651124 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:29.651127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:29.651167 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:29.651255 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:29.651798 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.652653 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.653615 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.653848 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.653962 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:29.655284 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:29.655287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:29.655289 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:29.655310 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:29.656862 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:29.657395 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:29.657398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:29.657399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:29.657472 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:29.657814 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:29.657817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:29.657818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:29.658059 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:29.658184 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:29.658512 (+ 328us) server_negotiation.cc:299] Negotiation successful
0807 06:22:29.658628 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260807 06:22:29.658875 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:29.650481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:44813 (local address 127.15.254.2:48721)
0807 06:22:29.650893 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:29.650908 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:29.651020 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:29.651276 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:29.651280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:29.651535 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:29.651653 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:29.651661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.652821 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.652825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:29.653481 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:29.653492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.653970 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.653974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:29.654076 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:29.654612 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:29.654624 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:29.655128 (+ 504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:29.656983 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:29.656990 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:29.656995 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:29.657284 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:29.657575 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:29.657581 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:29.657586 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:29.657705 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:29.658174 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:29.658178 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:29.658312 (+ 134us) client_negotiation.cc:769] Sending connection context
0807 06:22:29.658446 (+ 134us) client_negotiation.cc:240] Negotiation successful
0807 06:22:29.658644 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for kudu/127.15.254.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
I20260807 06:22:29.684078 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:0
--local_ip_for_outbound_sockets=127.15.254.3
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:44813
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:46841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:29.801364 19417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:29.801621 19417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:29.801704 19417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:29.805672 19417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:29.805727 19417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:29.805802 19417 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 06:22:29.805845 19417 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 06:22:29.805922 19417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:22:29.812728 19417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:46841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.254.62:44813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:29.813911 19417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:29.815003 19417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:29.825896 19425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:29.825867 19428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:29.826025 19417 server_base.cc:1034] running on GCE node
W20260807 06:22:29.826236 19423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:29.826691 19417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:29.827394 19417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:29.828577 19417 hybrid_clock.cc:648] HybridClock initialized: now 1786083749828531 us; error 48 us; skew 500 ppm
Aug 07 06:22:29 dist-test-slave-44d2 krb5kdc[18808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083749, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:29.831673 19417 init.cc:377] Logged in from keytab as kudu/127.15.254.3@KRBTEST.COM (short username kudu)
I20260807 06:22:29.832721 19417 webserver.cc:460] Webserver started at http://127.15.254.3:38739/ using document root <none> and password file <none>
I20260807 06:22:29.833384 19417 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:29.833483 19417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:29.833727 19417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:29.835871 19417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-2/data/instance:
uuid: "1e1ca1f8f92046da85f5100209b3209a"
format_stamp: "Formatted at 2026-08-07 06:22:29 on dist-test-slave-44d2"
server_key: "952daff5b5621cc30924cdb162cc647e"
server_key_iv: "808692e0859760644f98a5363665c38a"
server_key_version: "encryptionkey@0"
I20260807 06:22:29.836454 19417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance:
uuid: "1e1ca1f8f92046da85f5100209b3209a"
format_stamp: "Formatted at 2026-08-07 06:22:29 on dist-test-slave-44d2"
server_key: "952daff5b5621cc30924cdb162cc647e"
server_key_iv: "808692e0859760644f98a5363665c38a"
server_key_version: "encryptionkey@0"
I20260807 06:22:29.840204 19417 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:22:29.842869 19439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:29.844100 19417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 06:22:29.844239 19417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "1e1ca1f8f92046da85f5100209b3209a"
format_stamp: "Formatted at 2026-08-07 06:22:29 on dist-test-slave-44d2"
server_key: "952daff5b5621cc30924cdb162cc647e"
server_key_iv: "808692e0859760644f98a5363665c38a"
server_key_version: "encryptionkey@0"
I20260807 06:22:29.844354 19417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:29.877816 19417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:29.881394 19417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:29.881603 19417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:29.883122 19417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:29.883195 19417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:29.883270 19417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:29.883320 19417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 06:22:29 dist-test-slave-44d2 krb5kdc[18808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083749, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:29.894876 19417 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:33815
I20260807 06:22:29.894969 19593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:33815 every 8 connection(s)
I20260807 06:22:29.895895 19417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:22:29.898284 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:29.884100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:45929 (local address 127.15.254.62:44813)
0807 06:22:29.884238 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:29.884241 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:29.885356 (+ 1115us) server_negotiation.cc:407] Connection header received
0807 06:22:29.886549 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:29.886553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:29.886625 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:29.886744 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:29.888448 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.889369 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.890731 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.891123 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.891251 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:29.894089 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:29.894093 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:29.894096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:29.894124 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:29.895858 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:29.896416 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:29.896423 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:29.896425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:29.896576 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:29.897007 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:29.897010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:29.897012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:29.897294 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:29.897404 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:29.898013 (+ 609us) server_negotiation.cc:299] Negotiation successful
0807 06:22:29.898160 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 06:22:29.899138 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:29.884679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:44813 (local address 127.15.254.3:45929)
0807 06:22:29.885199 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:29.885245 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:29.886254 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:29.886989 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:29.886998 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:29.887498 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:29.888262 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:29.888283 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.889520 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.889524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:29.890592 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:29.890602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.891310 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.891315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:29.891546 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:29.892191 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:29.892208 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:29.893911 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:29.895982 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:29.895992 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:29.896010 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:29.896297 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:29.896677 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:29.896684 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:29.896687 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:29.896889 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:29.897410 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:29.897416 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:29.897767 (+ 351us) client_negotiation.cc:769] Sending connection context
0807 06:22:29.897936 (+ 169us) client_negotiation.cc:240] Negotiation successful
0807 06:22:29.898207 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":142,"threads_started":1}
I20260807 06:22:29.900450 19594 heartbeater.cc:347] Connected to a master server at 127.15.254.62:44813
I20260807 06:22:29.900624 19594 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:29.901232 19594 heartbeater.cc:511] Master 127.15.254.62:44813 requested a full tablet report, sending...
I20260807 06:22:29.901718 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 19417
I20260807 06:22:29.901859 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance
I20260807 06:22:29.902206 16376 external_mini_cluster.cc:1428] Setting key bf0785df9f4836e9230ee79b48e64e54
I20260807 06:22:29.902510 18882 ts_manager.cc:194] Registered new tserver with Master: 1e1ca1f8f92046da85f5100209b3209a (127.15.254.3:33815)
I20260807 06:22:29.903357 18882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.3@KRBTEST.COM'} at 127.15.254.3:45929
I20260807 06:22:29.904461 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:22:29.911582 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:29.902937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:46077 (local address 127.15.254.62:44813)
0807 06:22:29.903093 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:29.903099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:29.903857 (+ 758us) server_negotiation.cc:407] Connection header received
0807 06:22:29.903967 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:29.903970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:29.904015 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:29.904106 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:29.904684 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.905412 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.906768 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.906983 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.907140 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:29.908283 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:29.908289 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:29.908294 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:29.908327 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:29.909845 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:29.910395 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:29.910398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:29.910399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:29.910471 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:29.910785 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:29.910787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:29.910788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:29.910942 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:29.911054 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:29.911322 (+ 268us) server_negotiation.cc:299] Negotiation successful
0807 06:22:29.911429 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 06:22:29.911607 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:29.902881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:44813 (local address 127.15.254.3:46077)
0807 06:22:29.903765 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:29.903780 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:29.903878 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:29.904115 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:29.904117 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:29.904366 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:29.904554 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:29.904563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.905593 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.905599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:29.906637 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:29.906652 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.907140 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.907147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:29.907272 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:29.907828 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:29.907839 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:29.908141 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:29.909962 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:29.909969 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:29.909974 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:29.910297 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:29.910566 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:29.910571 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:29.910575 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:29.910687 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:29.911049 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:29.911052 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:29.911153 (+ 101us) client_negotiation.cc:769] Sending connection context
0807 06:22:29.911265 (+ 112us) client_negotiation.cc:240] Negotiation successful
0807 06:22:29.911419 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":143,"threads_started":1}
Aug 07 06:22:29 dist-test-slave-44d2 krb5kdc[18808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 06:22:29 dist-test-slave-44d2 krb5kdc[18808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:29.933565 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:29.925283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59972 (local address 127.15.254.62:44813)
0807 06:22:29.925421 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:29.925425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:29.925437 (+ 12us) server_negotiation.cc:407] Connection header received
0807 06:22:29.925480 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:29.925482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:29.925522 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:29.925594 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:29.926510 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.927044 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.928002 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.928217 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.928294 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:29.930664 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:29.930667 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:29.930669 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:29.930693 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:29.931931 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:29.932314 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:29.932316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:29.932317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:29.932393 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:29.932737 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:29.932740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:29.932741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:29.932934 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:29.933040 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:29.933245 (+ 205us) server_negotiation.cc:299] Negotiation successful
0807 06:22:29.933344 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 06:22:29.936686 18879 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:22:29.939909 18879 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:22:29.955106 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:29.949203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:33815 (local address 127.0.0.1:59112)
0807 06:22:29.950098 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:29.950166 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:29.950302 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:29.950833 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:29.950838 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:29.950863 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:29.951017 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:29.951024 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.952222 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.952226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:29.953564 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:29.953576 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.954461 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.954465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:29.954637 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:29.954684 (+ 47us) client_negotiation.cc:769] Sending connection context
0807 06:22:29.954794 (+ 110us) client_negotiation.cc:240] Negotiation successful
0807 06:22:29.954905 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":36,"thread_start_us":93,"threads_started":1}
I20260807 06:22:29.955111 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:29.949334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59112 (local address 127.15.254.3:33815)
0807 06:22:29.950300 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:29.950313 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:29.950342 (+ 29us) server_negotiation.cc:407] Connection header received
0807 06:22:29.950457 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:29.950463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:29.950662 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:29.950848 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:29.951137 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.952087 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.953719 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.954344 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.954457 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:29.954643 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:29.954845 (+ 202us) server_negotiation.cc:299] Negotiation successful
0807 06:22:29.954949 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":863,"thread_start_us":122,"threads_started":1}
I20260807 06:22:29.956092 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:29.949361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:39045 (local address 127.0.0.1:33956)
0807 06:22:29.950589 (+ 1228us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:29.950602 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:29.950683 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:29.951396 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:29.951400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:29.951418 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:29.951540 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:29.951549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.952984 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.952988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:29.954183 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:29.954194 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.955641 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.955644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:29.955746 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:29.955771 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 06:22:29.955861 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 06:22:29.955946 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1167,"mutex_wait_us":24,"thread_start_us":99,"threads_started":1}
I20260807 06:22:29.956627 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:29.949844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33956 (local address 127.15.254.2:39045)
0807 06:22:29.950933 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:29.950945 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:29.950967 (+ 22us) server_negotiation.cc:407] Connection header received
0807 06:22:29.951046 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:29.951054 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:29.951260 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:29.951417 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:29.951693 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.952854 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.954323 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.955152 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.955631 (+ 479us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:29.955795 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:29.955903 (+ 108us) server_negotiation.cc:299] Negotiation successful
0807 06:22:29.955993 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":992,"thread_start_us":127,"threads_started":1}
I20260807 06:22:29.956656 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:29.949611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:33979 (local address 127.0.0.1:50370)
0807 06:22:29.951655 (+ 2044us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:29.951670 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:29.951754 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:29.952146 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:29.952149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:29.952167 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:29.952298 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:29.952305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.953771 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.953774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:29.954906 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:29.954916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.956185 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.956189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:29.956288 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:29.956312 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 06:22:29.956387 (+ 75us) client_negotiation.cc:240] Negotiation successful
0807 06:22:29.956455 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1970,"mutex_wait_us":45,"thread_start_us":99,"threads_started":1}
I20260807 06:22:29.957304 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:29.950020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50370 (local address 127.15.254.1:33979)
0807 06:22:29.950319 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:29.950325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:29.951748 (+ 1423us) server_negotiation.cc:407] Connection header received
0807 06:22:29.951866 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:29.951871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:29.952021 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:29.952169 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:29.952398 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.953640 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.955040 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:29.955886 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:29.955978 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:29.956113 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:29.957044 (+ 931us) server_negotiation.cc:299] Negotiation successful
0807 06:22:29.957153 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":68,"threads_started":1}
I20260807 06:22:29.958566 19506 tablet_service.cc:1467] Processing CreateTablet for tablet 8ae7e409bca743258e77a509bfc7a229 (DEFAULT_TABLE table=test-table [id=bbbfb7e68a8c4162beb4f4b0524b3ee5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:29.959539 19506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae7e409bca743258e77a509bfc7a229. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:29.959574 19315 tablet_service.cc:1467] Processing CreateTablet for tablet 8ae7e409bca743258e77a509bfc7a229 (DEFAULT_TABLE table=test-table [id=bbbfb7e68a8c4162beb4f4b0524b3ee5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:29.960214 19100 tablet_service.cc:1467] Processing CreateTablet for tablet 8ae7e409bca743258e77a509bfc7a229 (DEFAULT_TABLE table=test-table [id=bbbfb7e68a8c4162beb4f4b0524b3ee5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:29.960706 19315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae7e409bca743258e77a509bfc7a229. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:29.961243 19100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae7e409bca743258e77a509bfc7a229. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:29.966918 19623 tablet_bootstrap.cc:492] T 8ae7e409bca743258e77a509bfc7a229 P 1e1ca1f8f92046da85f5100209b3209a: Bootstrap starting.
I20260807 06:22:29.969310 19623 tablet_bootstrap.cc:654] T 8ae7e409bca743258e77a509bfc7a229 P 1e1ca1f8f92046da85f5100209b3209a: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:29.969404 19624 tablet_bootstrap.cc:492] T 8ae7e409bca743258e77a509bfc7a229 P f4b98f4bd4d64a549306c2abd1f0851a: Bootstrap starting.
I20260807 06:22:29.970836 19623 log.cc:824] T 8ae7e409bca743258e77a509bfc7a229 P 1e1ca1f8f92046da85f5100209b3209a: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:29.971668 19624 tablet_bootstrap.cc:654] T 8ae7e409bca743258e77a509bfc7a229 P f4b98f4bd4d64a549306c2abd1f0851a: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:29.972087 19625 tablet_bootstrap.cc:492] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf: Bootstrap starting.
I20260807 06:22:29.972592 19624 log.cc:824] T 8ae7e409bca743258e77a509bfc7a229 P f4b98f4bd4d64a549306c2abd1f0851a: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:29.973308 19623 tablet_bootstrap.cc:492] T 8ae7e409bca743258e77a509bfc7a229 P 1e1ca1f8f92046da85f5100209b3209a: No bootstrap required, opened a new log
I20260807 06:22:29.973522 19623 ts_tablet_manager.cc:1397] T 8ae7e409bca743258e77a509bfc7a229 P 1e1ca1f8f92046da85f5100209b3209a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260807 06:22:29.974246 19625 tablet_bootstrap.cc:654] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:29.974619 19624 tablet_bootstrap.cc:492] T 8ae7e409bca743258e77a509bfc7a229 P f4b98f4bd4d64a549306c2abd1f0851a: No bootstrap required, opened a new log
I20260807 06:22:29.974861 19624 ts_tablet_manager.cc:1397] T 8ae7e409bca743258e77a509bfc7a229 P f4b98f4bd4d64a549306c2abd1f0851a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 06:22:29.975076 19625 log.cc:824] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:29.977305 19625 tablet_bootstrap.cc:492] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf: No bootstrap required, opened a new log
I20260807 06:22:29.977245 19623 raft_consensus.cc:348] T 8ae7e409bca743258e77a509bfc7a229 P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } }
I20260807 06:22:29.977510 19625 ts_tablet_manager.cc:1397] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 06:22:29.977555 19623 raft_consensus.cc:374] T 8ae7e409bca743258e77a509bfc7a229 P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:29.977612 19623 raft_consensus.cc:729] T 8ae7e409bca743258e77a509bfc7a229 P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e1ca1f8f92046da85f5100209b3209a, State: Initialized, Role: FOLLOWER
I20260807 06:22:29.978101 19623 consensus_queue.cc:260] T 8ae7e409bca743258e77a509bfc7a229 P 1e1ca1f8f92046da85f5100209b3209a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } }
I20260807 06:22:29.979183 19623 ts_tablet_manager.cc:1428] T 8ae7e409bca743258e77a509bfc7a229 P 1e1ca1f8f92046da85f5100209b3209a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260807 06:22:29.979605 19594 heartbeater.cc:503] Master 127.15.254.62:44813 was elected leader, sending a full tablet report...
I20260807 06:22:29.979538 19624 raft_consensus.cc:348] T 8ae7e409bca743258e77a509bfc7a229 P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } }
I20260807 06:22:29.979825 19624 raft_consensus.cc:374] T 8ae7e409bca743258e77a509bfc7a229 P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:29.979895 19624 raft_consensus.cc:729] T 8ae7e409bca743258e77a509bfc7a229 P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b98f4bd4d64a549306c2abd1f0851a, State: Initialized, Role: FOLLOWER
I20260807 06:22:29.980358 19624 consensus_queue.cc:260] T 8ae7e409bca743258e77a509bfc7a229 P f4b98f4bd4d64a549306c2abd1f0851a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } }
I20260807 06:22:29.981123 19203 heartbeater.cc:503] Master 127.15.254.62:44813 was elected leader, sending a full tablet report...
I20260807 06:22:29.981419 19624 ts_tablet_manager.cc:1428] T 8ae7e409bca743258e77a509bfc7a229 P f4b98f4bd4d64a549306c2abd1f0851a: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260807 06:22:29.982861 19625 raft_consensus.cc:348] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } }
I20260807 06:22:29.983129 19625 raft_consensus.cc:374] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:29.983191 19625 raft_consensus.cc:729] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2c6d77d1d7c4b1888a2d1385067cedf, State: Initialized, Role: FOLLOWER
I20260807 06:22:29.983628 19625 consensus_queue.cc:260] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } }
I20260807 06:22:29.984673 19625 ts_tablet_manager.cc:1428] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260807 06:22:29.984679 19407 heartbeater.cc:503] Master 127.15.254.62:44813 was elected leader, sending a full tablet report...
I20260807 06:22:30.050194 19636 raft_consensus.cc:482] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:22:30.050436 19636 raft_consensus.cc:504] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } }
I20260807 06:22:30.051599 19636 leader_election.cc:288] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e1ca1f8f92046da85f5100209b3209a (127.15.254.3:33815), f4b98f4bd4d64a549306c2abd1f0851a (127.15.254.1:33979)
I20260807 06:22:30.056100 19252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:30.051883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:33815 (local address 127.15.254.2:56151)
0807 06:22:30.052119 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:30.052138 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:30.052237 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:30.052738 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:30.052745 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:30.052769 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:30.052951 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:30.052958 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.053818 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.053822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:30.054999 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:30.055012 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.055741 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.055745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:30.055833 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:30.055855 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 06:22:30.055924 (+ 69us) client_negotiation.cc:240] Negotiation successful
0807 06:22:30.055984 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":1}
I20260807 06:22:30.056211 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:30.052166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:56151 (local address 127.15.254.3:33815)
0807 06:22:30.052304 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:30.052310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:30.052325 (+ 15us) server_negotiation.cc:407] Connection header received
0807 06:22:30.052476 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:30.052542 (+ 66us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:30.052623 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:30.052710 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:30.053115 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.053664 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.055156 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.055598 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.055730 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:30.055795 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:30.055967 (+ 172us) server_negotiation.cc:299] Negotiation successful
0807 06:22:30.056081 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 06:22:30.056874 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:30.051883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:33979 (local address 127.15.254.2:33015)
0807 06:22:30.052458 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:30.052475 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:30.052709 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:30.053077 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:30.053082 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:30.053107 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:30.053618 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:30.053625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.054424 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.054429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:30.055504 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:30.055514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.056460 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.056465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:30.056601 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:30.056632 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 06:22:30.056707 (+ 75us) client_negotiation.cc:240] Negotiation successful
0807 06:22:30.056762 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":33,"thread_start_us":170,"threads_started":1}
I20260807 06:22:30.056957 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:30.051990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:33015 (local address 127.15.254.1:33979)
0807 06:22:30.052149 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:30.052154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:30.052640 (+ 486us) server_negotiation.cc:407] Connection header received
0807 06:22:30.052911 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:30.052914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:30.052958 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:30.053024 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:30.053767 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.054285 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.055656 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.056284 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.056417 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:30.056467 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:30.056751 (+ 284us) server_negotiation.cc:299] Negotiation successful
0807 06:22:30.056837 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 06:22:30.056932 19532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae7e409bca743258e77a509bfc7a229" candidate_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e1ca1f8f92046da85f5100209b3209a" is_pre_election: true
I20260807 06:22:30.057291 19532 raft_consensus.cc:2454] T 8ae7e409bca743258e77a509bfc7a229 P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2c6d77d1d7c4b1888a2d1385067cedf in term 0.
I20260807 06:22:30.057583 19127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae7e409bca743258e77a509bfc7a229" candidate_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" is_pre_election: true
I20260807 06:22:30.057928 19127 raft_consensus.cc:2454] T 8ae7e409bca743258e77a509bfc7a229 P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2c6d77d1d7c4b1888a2d1385067cedf in term 0.
I20260807 06:22:30.057909 19243 leader_election.cc:302] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e1ca1f8f92046da85f5100209b3209a, c2c6d77d1d7c4b1888a2d1385067cedf; no voters:
I20260807 06:22:30.058260 19636 raft_consensus.cc:2781] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:22:30.058369 19636 raft_consensus.cc:482] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:22:30.058420 19636 raft_consensus.cc:3037] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:30.059643 19636 raft_consensus.cc:504] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } }
I20260807 06:22:30.059979 19636 leader_election.cc:288] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf [CANDIDATE]: Term 1 election: Requested vote from peers 1e1ca1f8f92046da85f5100209b3209a (127.15.254.3:33815), f4b98f4bd4d64a549306c2abd1f0851a (127.15.254.1:33979)
I20260807 06:22:30.060367 19532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae7e409bca743258e77a509bfc7a229" candidate_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e1ca1f8f92046da85f5100209b3209a"
I20260807 06:22:30.060432 19127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae7e409bca743258e77a509bfc7a229" candidate_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b98f4bd4d64a549306c2abd1f0851a"
I20260807 06:22:30.060555 19532 raft_consensus.cc:3037] T 8ae7e409bca743258e77a509bfc7a229 P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:30.060629 19127 raft_consensus.cc:3037] T 8ae7e409bca743258e77a509bfc7a229 P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:30.061647 19532 raft_consensus.cc:2454] T 8ae7e409bca743258e77a509bfc7a229 P 1e1ca1f8f92046da85f5100209b3209a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2c6d77d1d7c4b1888a2d1385067cedf in term 1.
I20260807 06:22:30.061853 19127 raft_consensus.cc:2454] T 8ae7e409bca743258e77a509bfc7a229 P f4b98f4bd4d64a549306c2abd1f0851a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2c6d77d1d7c4b1888a2d1385067cedf in term 1.
I20260807 06:22:30.062008 19243 leader_election.cc:302] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e1ca1f8f92046da85f5100209b3209a, c2c6d77d1d7c4b1888a2d1385067cedf; no voters:
I20260807 06:22:30.062224 19636 raft_consensus.cc:2781] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:30.062471 19636 raft_consensus.cc:686] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf [term 1 LEADER]: Becoming Leader. State: Replica: c2c6d77d1d7c4b1888a2d1385067cedf, State: Running, Role: LEADER
I20260807 06:22:30.062822 19636 consensus_queue.cc:237] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } }
I20260807 06:22:30.066416 18879 catalog_manager.cc:5495] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf reported cstate change: term changed from 0 to 1, leader changed from <none> to c2c6d77d1d7c4b1888a2d1385067cedf (127.15.254.2). New cstate: current_term: 1 leader_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } health_report { overall_health: UNKNOWN } } }
Aug 07 06:22:30 dist-test-slave-44d2 krb5kdc[18808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083749, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:30.098984 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:30.090141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59974 (local address 127.15.254.62:44813)
0807 06:22:30.090301 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:30.090306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:30.090444 (+ 138us) server_negotiation.cc:407] Connection header received
0807 06:22:30.090619 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:30.090622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:30.090667 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:30.090752 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:30.091380 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.092113 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.093056 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.093259 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.093370 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:30.095528 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:30.095532 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:30.095534 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:30.095565 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:30.097141 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:30.097622 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:30.097627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:30.097629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:30.097711 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:30.097981 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:30.097985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:30.097987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:30.098221 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:30.098332 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:30.098638 (+ 306us) server_negotiation.cc:299] Negotiation successful
0807 06:22:30.098846 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":72}
I20260807 06:22:30.098984 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:30.090054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:44813 (local address 127.0.0.1:59974)
0807 06:22:30.090356 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:30.090369 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:30.090500 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:30.090773 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:30.090776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:30.091167 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:30.091258 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:30.091264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.092261 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.092266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:30.092901 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:30.092911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.093371 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.093374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:30.093448 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:30.093986 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:30.093998 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:30.095395 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:30.097245 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:30.097258 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:30.097264 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:30.097511 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:30.097800 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:30.097805 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:30.097807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:30.097893 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:30.098332 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:30.098341 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:30.098461 (+ 120us) client_negotiation.cc:769] Sending connection context
0807 06:22:30.098590 (+ 129us) client_negotiation.cc:240] Negotiation successful
0807 06:22:30.098780 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":111,"threads_started":1}
I20260807 06:22:30.103353 18879 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:59974:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 06:22:30.104017 18879 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:22:30.109169 19100 tablet_service.cc:1467] Processing CreateTablet for tablet cbe9cfff474a4d9283141591d465bedf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d35f9200509544c7b3167c37af72402c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 06:22:30.109205 19315 tablet_service.cc:1467] Processing CreateTablet for tablet cbe9cfff474a4d9283141591d465bedf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d35f9200509544c7b3167c37af72402c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 06:22:30.109541 19506 tablet_service.cc:1467] Processing CreateTablet for tablet cbe9cfff474a4d9283141591d465bedf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d35f9200509544c7b3167c37af72402c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 06:22:30.109992 19100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbe9cfff474a4d9283141591d465bedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:30.110000 19315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbe9cfff474a4d9283141591d465bedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:30.110143 19506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbe9cfff474a4d9283141591d465bedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:30.112922 19625 tablet_bootstrap.cc:492] T cbe9cfff474a4d9283141591d465bedf P c2c6d77d1d7c4b1888a2d1385067cedf: Bootstrap starting.
I20260807 06:22:30.113073 19623 tablet_bootstrap.cc:492] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a: Bootstrap starting.
I20260807 06:22:30.113773 19625 tablet_bootstrap.cc:654] T cbe9cfff474a4d9283141591d465bedf P c2c6d77d1d7c4b1888a2d1385067cedf: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:30.114025 19623 tablet_bootstrap.cc:654] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:30.114786 19624 tablet_bootstrap.cc:492] T cbe9cfff474a4d9283141591d465bedf P f4b98f4bd4d64a549306c2abd1f0851a: Bootstrap starting.
I20260807 06:22:30.115307 19625 tablet_bootstrap.cc:492] T cbe9cfff474a4d9283141591d465bedf P c2c6d77d1d7c4b1888a2d1385067cedf: No bootstrap required, opened a new log
I20260807 06:22:30.115410 19625 ts_tablet_manager.cc:1397] T cbe9cfff474a4d9283141591d465bedf P c2c6d77d1d7c4b1888a2d1385067cedf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 06:22:30.115691 19624 tablet_bootstrap.cc:654] T cbe9cfff474a4d9283141591d465bedf P f4b98f4bd4d64a549306c2abd1f0851a: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:30.115854 19625 raft_consensus.cc:348] T cbe9cfff474a4d9283141591d465bedf P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:30.115963 19625 raft_consensus.cc:374] T cbe9cfff474a4d9283141591d465bedf P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:30.116007 19625 raft_consensus.cc:729] T cbe9cfff474a4d9283141591d465bedf P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2c6d77d1d7c4b1888a2d1385067cedf, State: Initialized, Role: FOLLOWER
I20260807 06:22:30.116165 19625 consensus_queue.cc:260] T cbe9cfff474a4d9283141591d465bedf P c2c6d77d1d7c4b1888a2d1385067cedf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:30.116622 19625 ts_tablet_manager.cc:1428] T cbe9cfff474a4d9283141591d465bedf P c2c6d77d1d7c4b1888a2d1385067cedf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 06:22:30.117004 19624 tablet_bootstrap.cc:492] T cbe9cfff474a4d9283141591d465bedf P f4b98f4bd4d64a549306c2abd1f0851a: No bootstrap required, opened a new log
I20260807 06:22:30.117074 19624 ts_tablet_manager.cc:1397] T cbe9cfff474a4d9283141591d465bedf P f4b98f4bd4d64a549306c2abd1f0851a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260807 06:22:30.117214 19623 tablet_bootstrap.cc:492] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a: No bootstrap required, opened a new log
I20260807 06:22:30.117292 19623 ts_tablet_manager.cc:1397] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260807 06:22:30.117478 19624 raft_consensus.cc:348] T cbe9cfff474a4d9283141591d465bedf P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:30.117591 19624 raft_consensus.cc:374] T cbe9cfff474a4d9283141591d465bedf P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:30.117615 19624 raft_consensus.cc:729] T cbe9cfff474a4d9283141591d465bedf P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b98f4bd4d64a549306c2abd1f0851a, State: Initialized, Role: FOLLOWER
I20260807 06:22:30.117743 19624 consensus_queue.cc:260] T cbe9cfff474a4d9283141591d465bedf P f4b98f4bd4d64a549306c2abd1f0851a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:30.117810 19623 raft_consensus.cc:348] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:30.117923 19623 raft_consensus.cc:374] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:30.117957 19623 raft_consensus.cc:729] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e1ca1f8f92046da85f5100209b3209a, State: Initialized, Role: FOLLOWER
I20260807 06:22:30.118023 19624 ts_tablet_manager.cc:1428] T cbe9cfff474a4d9283141591d465bedf P f4b98f4bd4d64a549306c2abd1f0851a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260807 06:22:30.117991 19635 tablet_replica.cc:439] T cbe9cfff474a4d9283141591d465bedf P f4b98f4bd4d64a549306c2abd1f0851a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } } }
I20260807 06:22:30.118137 19635 tablet_replica.cc:442] T cbe9cfff474a4d9283141591d465bedf P f4b98f4bd4d64a549306c2abd1f0851a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:22:30.118098 19623 consensus_queue.cc:260] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:30.118400 19623 ts_tablet_manager.cc:1428] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260807 06:22:30.118392 19632 tablet_replica.cc:439] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } } }
I20260807 06:22:30.118590 19632 tablet_replica.cc:442] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:22:30.116449 19636 tablet_replica.cc:439] T cbe9cfff474a4d9283141591d465bedf P c2c6d77d1d7c4b1888a2d1385067cedf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } } }
I20260807 06:22:30.118701 19636 tablet_replica.cc:442] T cbe9cfff474a4d9283141591d465bedf P c2c6d77d1d7c4b1888a2d1385067cedf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:22:30.132841 19632 raft_consensus.cc:482] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:22:30.133010 19632 raft_consensus.cc:504] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:30.134366 19632 leader_election.cc:288] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4b98f4bd4d64a549306c2abd1f0851a (127.15.254.1:33979), c2c6d77d1d7c4b1888a2d1385067cedf (127.15.254.2:39045)
I20260807 06:22:30.138046 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:30.134695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:53777 (local address 127.15.254.1:33979)
0807 06:22:30.134847 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:30.134852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:30.134869 (+ 17us) server_negotiation.cc:407] Connection header received
0807 06:22:30.134923 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:30.134927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:30.134976 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:30.135077 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:30.135413 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.135929 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.137013 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.137441 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.137601 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:30.137660 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:30.137844 (+ 184us) server_negotiation.cc:299] Negotiation successful
0807 06:22:30.137924 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 06:22:30.138072 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:30.134557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:33979 (local address 127.15.254.3:53777)
0807 06:22:30.134737 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:30.134754 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:30.134840 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:30.135134 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:30.135139 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:30.135161 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:30.135292 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:30.135299 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.136077 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.136080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:30.136882 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:30.136891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.137601 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.137605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:30.137720 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:30.137738 (+ 18us) client_negotiation.cc:769] Sending connection context
0807 06:22:30.137826 (+ 88us) client_negotiation.cc:240] Negotiation successful
0807 06:22:30.137903 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260807 06:22:30.138602 19127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbe9cfff474a4d9283141591d465bedf" candidate_uuid: "1e1ca1f8f92046da85f5100209b3209a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" is_pre_election: true
I20260807 06:22:30.138763 19127 raft_consensus.cc:2454] T cbe9cfff474a4d9283141591d465bedf P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e1ca1f8f92046da85f5100209b3209a in term 0.
I20260807 06:22:30.139250 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:30.134560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:39045 (local address 127.15.254.3:56001)
0807 06:22:30.135391 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:30.135404 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:30.135531 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:30.135818 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:30.135821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:30.135840 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:30.135990 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:30.135996 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.136895 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.136901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:30.137830 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:30.137840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.138759 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.138764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:30.138880 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:30.138909 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 06:22:30.139020 (+ 111us) client_negotiation.cc:240] Negotiation successful
0807 06:22:30.139105 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":275,"thread_start_us":270,"threads_started":1}
I20260807 06:22:30.139281 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:30.134710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:56001 (local address 127.15.254.2:39045)
0807 06:22:30.134819 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:30.134824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:30.135501 (+ 677us) server_negotiation.cc:407] Connection header received
0807 06:22:30.135656 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:30.135666 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:30.135708 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:30.135801 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:30.136115 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.136764 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.137975 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.138621 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.138768 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:30.138822 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:30.139075 (+ 253us) server_negotiation.cc:299] Negotiation successful
0807 06:22:30.139157 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260807 06:22:30.139215 19440 leader_election.cc:302] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e1ca1f8f92046da85f5100209b3209a, f4b98f4bd4d64a549306c2abd1f0851a; no voters:
I20260807 06:22:30.139544 19632 raft_consensus.cc:2781] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:22:30.139626 19632 raft_consensus.cc:482] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:22:30.139690 19632 raft_consensus.cc:3037] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:30.139922 19344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbe9cfff474a4d9283141591d465bedf" candidate_uuid: "1e1ca1f8f92046da85f5100209b3209a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" is_pre_election: true
I20260807 06:22:30.140172 19344 raft_consensus.cc:2454] T cbe9cfff474a4d9283141591d465bedf P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e1ca1f8f92046da85f5100209b3209a in term 0.
I20260807 06:22:30.141121 19632 raft_consensus.cc:504] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:30.141573 19632 leader_election.cc:288] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a [CANDIDATE]: Term 1 election: Requested vote from peers f4b98f4bd4d64a549306c2abd1f0851a (127.15.254.1:33979), c2c6d77d1d7c4b1888a2d1385067cedf (127.15.254.2:39045)
I20260807 06:22:30.141901 19127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbe9cfff474a4d9283141591d465bedf" candidate_uuid: "1e1ca1f8f92046da85f5100209b3209a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b98f4bd4d64a549306c2abd1f0851a"
I20260807 06:22:30.142009 19127 raft_consensus.cc:3037] T cbe9cfff474a4d9283141591d465bedf P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:30.142963 19127 raft_consensus.cc:2454] T cbe9cfff474a4d9283141591d465bedf P f4b98f4bd4d64a549306c2abd1f0851a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e1ca1f8f92046da85f5100209b3209a in term 1.
I20260807 06:22:30.143214 19344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbe9cfff474a4d9283141591d465bedf" candidate_uuid: "1e1ca1f8f92046da85f5100209b3209a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf"
I20260807 06:22:30.143242 19440 leader_election.cc:302] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e1ca1f8f92046da85f5100209b3209a, f4b98f4bd4d64a549306c2abd1f0851a; no voters:
I20260807 06:22:30.143339 19344 raft_consensus.cc:3037] T cbe9cfff474a4d9283141591d465bedf P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:30.143429 19632 raft_consensus.cc:2781] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:30.143635 19632 raft_consensus.cc:686] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a [term 1 LEADER]: Becoming Leader. State: Replica: 1e1ca1f8f92046da85f5100209b3209a, State: Running, Role: LEADER
I20260807 06:22:30.143975 19632 consensus_queue.cc:237] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
W20260807 06:22:30.144296 19408 tablet.cc:2369] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:22:30.144604 19344 raft_consensus.cc:2454] T cbe9cfff474a4d9283141591d465bedf P c2c6d77d1d7c4b1888a2d1385067cedf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e1ca1f8f92046da85f5100209b3209a in term 1.
I20260807 06:22:30.146438 19653 tablet_replica.cc:439] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a: TxnStatusTablet state changed. Reason: New leader 1e1ca1f8f92046da85f5100209b3209a. Latest consensus state: current_term: 1 leader_uuid: "1e1ca1f8f92046da85f5100209b3209a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } } }
I20260807 06:22:30.146554 19653 tablet_replica.cc:442] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a: This TxnStatusTablet replica's current role is: LEADER
I20260807 06:22:30.146885 19656 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 06:22:30.147159 19656 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260807 06:22:30.147410 19596 tablet.cc:2369] T 8ae7e409bca743258e77a509bfc7a229 P 1e1ca1f8f92046da85f5100209b3209a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:22:30.147753 18882 catalog_manager.cc:5495] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e1ca1f8f92046da85f5100209b3209a (127.15.254.3). New cstate: current_term: 1 leader_uuid: "1e1ca1f8f92046da85f5100209b3209a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } health_report { overall_health: HEALTHY } } }
W20260807 06:22:30.148208 19205 tablet.cc:2369] T 8ae7e409bca743258e77a509bfc7a229 P f4b98f4bd4d64a549306c2abd1f0851a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:22:30.160753 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:30.156743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59114 (local address 127.15.254.3:33815)
0807 06:22:30.156878 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:30.156881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:30.156895 (+ 14us) server_negotiation.cc:407] Connection header received
0807 06:22:30.157077 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:30.157079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:30.157128 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:30.157210 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:22:30.157489 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.158001 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.159267 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.159540 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.159622 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:30.159950 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:22:30.160326 (+ 376us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:22:30.160412 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:30.160605 (+ 193us) server_negotiation.cc:299] Negotiation successful
0807 06:22:30.160652 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 06:22:30.160822 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:30.156636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:33815 (local address 127.0.0.1:59114)
0807 06:22:30.156801 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:30.156817 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:30.156927 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:30.157258 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:30.157261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:30.157270 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 06:22:30.157379 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:30.157385 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.158119 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.158124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:30.159108 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:30.159120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.159659 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.159663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:30.159756 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:30.159811 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 06:22:30.160429 (+ 618us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 06:22:30.160454 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 06:22:30.160592 (+ 138us) client_negotiation.cc:240] Negotiation successful
0807 06:22:30.160683 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 06:22:30.165874 19344 raft_consensus.cc:1261] T cbe9cfff474a4d9283141591d465bedf P c2c6d77d1d7c4b1888a2d1385067cedf [term 1 FOLLOWER]: Refusing update from remote peer 1e1ca1f8f92046da85f5100209b3209a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:22:30.165875 19127 raft_consensus.cc:1261] T cbe9cfff474a4d9283141591d465bedf P f4b98f4bd4d64a549306c2abd1f0851a [term 1 FOLLOWER]: Refusing update from remote peer 1e1ca1f8f92046da85f5100209b3209a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:22:30.166498 19632 consensus_queue.cc:1035] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:22:30.166671 19653 consensus_queue.cc:1035] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:22:30.169826 19636 tablet_replica.cc:439] T cbe9cfff474a4d9283141591d465bedf P c2c6d77d1d7c4b1888a2d1385067cedf: TxnStatusTablet state changed. Reason: New leader 1e1ca1f8f92046da85f5100209b3209a. Latest consensus state: current_term: 1 leader_uuid: "1e1ca1f8f92046da85f5100209b3209a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } } }
I20260807 06:22:30.169941 19636 tablet_replica.cc:442] T cbe9cfff474a4d9283141591d465bedf P c2c6d77d1d7c4b1888a2d1385067cedf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:22:30.170092 19635 tablet_replica.cc:439] T cbe9cfff474a4d9283141591d465bedf P f4b98f4bd4d64a549306c2abd1f0851a: TxnStatusTablet state changed. Reason: New leader 1e1ca1f8f92046da85f5100209b3209a. Latest consensus state: current_term: 1 leader_uuid: "1e1ca1f8f92046da85f5100209b3209a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } } }
I20260807 06:22:30.170219 19635 tablet_replica.cc:442] T cbe9cfff474a4d9283141591d465bedf P f4b98f4bd4d64a549306c2abd1f0851a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:22:30.173676 19635 tablet_replica.cc:439] T cbe9cfff474a4d9283141591d465bedf P f4b98f4bd4d64a549306c2abd1f0851a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e1ca1f8f92046da85f5100209b3209a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } } }
I20260807 06:22:30.173801 19635 tablet_replica.cc:442] T cbe9cfff474a4d9283141591d465bedf P f4b98f4bd4d64a549306c2abd1f0851a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:22:30.175015 19632 tablet_replica.cc:439] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e1ca1f8f92046da85f5100209b3209a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } } }
I20260807 06:22:30.175128 19632 tablet_replica.cc:442] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a: This TxnStatusTablet replica's current role is: LEADER
I20260807 06:22:30.183629 19632 tablet_replica.cc:439] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e1ca1f8f92046da85f5100209b3209a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } } }
I20260807 06:22:30.183769 19632 tablet_replica.cc:442] T cbe9cfff474a4d9283141591d465bedf P 1e1ca1f8f92046da85f5100209b3209a: This TxnStatusTablet replica's current role is: LEADER
I20260807 06:22:30.184640 19636 tablet_replica.cc:439] T cbe9cfff474a4d9283141591d465bedf P c2c6d77d1d7c4b1888a2d1385067cedf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e1ca1f8f92046da85f5100209b3209a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } } }
I20260807 06:22:30.184775 19636 tablet_replica.cc:442] T cbe9cfff474a4d9283141591d465bedf P c2c6d77d1d7c4b1888a2d1385067cedf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:22:30.186247 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:30.180702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33972 (local address 127.15.254.2:39045)
0807 06:22:30.180826 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:30.180830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:30.180843 (+ 13us) server_negotiation.cc:407] Connection header received
0807 06:22:30.181056 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:30.181059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:30.181115 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:30.181208 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:22:30.181464 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.181958 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.183149 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.183491 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.183600 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:30.184725 (+ 1125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:22:30.185243 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:22:30.185360 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:30.186065 (+ 705us) server_negotiation.cc:299] Negotiation successful
0807 06:22:30.186124 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 06:22:30.197724 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:30.193697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:60939 (local address 127.15.254.3:33815)
0807 06:22:30.193831 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:30.193837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:30.194169 (+ 332us) server_negotiation.cc:407] Connection header received
0807 06:22:30.194296 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:30.194300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:30.194357 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:30.194442 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:22:30.194689 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.195467 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.196405 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.196803 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.196911 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:30.197249 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:22:30.197331 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:22:30.197413 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:30.197533 (+ 120us) server_negotiation.cc:299] Negotiation successful
0807 06:22:30.197585 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 06:22:30.197749 19678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:30.193565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:33815 (local address 127.15.254.2:60939)
0807 06:22:30.194067 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:30.194083 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:30.194191 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:30.194460 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:30.194463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:30.194469 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 06:22:30.194585 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:30.194589 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.195603 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.195606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:30.196290 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:30.196300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.196931 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.196934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:30.197003 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:30.197051 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 06:22:30.197418 (+ 367us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 06:22:30.197427 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 06:22:30.197518 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 06:22:30.197609 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":107,"threads_started":1}
I20260807 06:22:30.204928 19127 raft_consensus.cc:1261] T 8ae7e409bca743258e77a509bfc7a229 P f4b98f4bd4d64a549306c2abd1f0851a [term 1 FOLLOWER]: Refusing update from remote peer c2c6d77d1d7c4b1888a2d1385067cedf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:22:30.205322 19532 raft_consensus.cc:1261] T 8ae7e409bca743258e77a509bfc7a229 P 1e1ca1f8f92046da85f5100209b3209a [term 1 FOLLOWER]: Refusing update from remote peer c2c6d77d1d7c4b1888a2d1385067cedf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:22:30.205381 19636 consensus_queue.cc:1035] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:22:30.205920 19636 consensus_queue.cc:1035] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:22:30.210158 19665 mvcc.cc:205] Tried to move back new op lower bound from 7315799040834981888 to 7315799040260640768. Current Snapshot: MvccSnapshot[applied={T|T < 7315799040834981888}]
I20260807 06:22:30.233906 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:30.230334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:57755 (local address 127.15.254.2:39045)
0807 06:22:30.230454 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:30.230459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:30.230473 (+ 14us) server_negotiation.cc:407] Connection header received
0807 06:22:30.230517 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:30.230519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:30.230564 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:30.230652 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:22:30.230911 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.231623 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.232801 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.233031 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.233139 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:30.233471 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:22:30.233547 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:22:30.233628 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:30.233718 (+ 90us) server_negotiation.cc:299] Negotiation successful
0807 06:22:30.233768 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 06:22:30.233906 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:30.229917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:39045 (local address 127.15.254.3:57755)
0807 06:22:30.230133 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:30.230146 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:30.230218 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:30.230667 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:30.230670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:30.230678 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 06:22:30.230804 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:30.230809 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.231822 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.231824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:30.232685 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:30.232696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:30.233140 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:30.233143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:30.233250 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:30.233297 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 06:22:30.233634 (+ 337us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 06:22:30.233643 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 06:22:30.233708 (+ 65us) client_negotiation.cc:240] Negotiation successful
0807 06:22:30.233787 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260807 06:22:30.264760 18881 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:22:30.264992 18881 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:22:30.268067 18881 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce: Sending DeleteTablet for 3 replicas of tablet 8ae7e409bca743258e77a509bfc7a229
I20260807 06:22:30.268905 19506 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ae7e409bca743258e77a509bfc7a229 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:30 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:59112
I20260807 06:22:30.269111 19315 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ae7e409bca743258e77a509bfc7a229 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:30 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:33956
I20260807 06:22:30.269635 19686 tablet_replica.cc:330] T 8ae7e409bca743258e77a509bfc7a229 P 1e1ca1f8f92046da85f5100209b3209a: stopping tablet replica
I20260807 06:22:30.269955 19686 raft_consensus.cc:2229] T 8ae7e409bca743258e77a509bfc7a229 P 1e1ca1f8f92046da85f5100209b3209a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:22:30.269982 19100 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ae7e409bca743258e77a509bfc7a229 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:30 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:50370
I20260807 06:22:30.270151 19686 raft_consensus.cc:2258] T 8ae7e409bca743258e77a509bfc7a229 P 1e1ca1f8f92046da85f5100209b3209a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:30.270666 19688 tablet_replica.cc:330] T 8ae7e409bca743258e77a509bfc7a229 P f4b98f4bd4d64a549306c2abd1f0851a: stopping tablet replica
I20260807 06:22:30.270983 19688 raft_consensus.cc:2229] T 8ae7e409bca743258e77a509bfc7a229 P f4b98f4bd4d64a549306c2abd1f0851a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:22:30.271239 19688 raft_consensus.cc:2258] T 8ae7e409bca743258e77a509bfc7a229 P f4b98f4bd4d64a549306c2abd1f0851a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:30.271476 18881 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 06:22:30.271622 19687 tablet_replica.cc:330] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf: stopping tablet replica
I20260807 06:22:30.271901 19687 raft_consensus.cc:2229] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf [term 1 LEADER]: Raft consensus shutting down.
W20260807 06:22:30.271955 18881 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:22:30.272293 19687 raft_consensus.cc:2258] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:30.274115 19688 ts_tablet_manager.cc:1905] T 8ae7e409bca743258e77a509bfc7a229 P f4b98f4bd4d64a549306c2abd1f0851a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:22:30.276216 19687 ts_tablet_manager.cc:1905] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:22:30.277781 19686 ts_tablet_manager.cc:1905] T 8ae7e409bca743258e77a509bfc7a229 P 1e1ca1f8f92046da85f5100209b3209a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:22:30.277741 19315 tablet_service.cc:1467] Processing CreateTablet for tablet 21dd9d873c0242d4aa3c3471a5edab0c (DEFAULT_TABLE table=test-table [id=b2321333edbf4342a256114de7a243b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:30.278146 19315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21dd9d873c0242d4aa3c3471a5edab0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:30.279384 19687 ts_tablet_manager.cc:1918] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:22:30.279488 19687 log.cc:1196] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/8ae7e409bca743258e77a509bfc7a229
I20260807 06:22:30.279831 19687 ts_tablet_manager.cc:1939] T 8ae7e409bca743258e77a509bfc7a229 P c2c6d77d1d7c4b1888a2d1385067cedf: Deleting consensus metadata
I20260807 06:22:30.280761 19506 tablet_service.cc:1467] Processing CreateTablet for tablet 21dd9d873c0242d4aa3c3471a5edab0c (DEFAULT_TABLE table=test-table [id=b2321333edbf4342a256114de7a243b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:30.281064 19506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21dd9d873c0242d4aa3c3471a5edab0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:30.283016 19688 ts_tablet_manager.cc:1918] T 8ae7e409bca743258e77a509bfc7a229 P f4b98f4bd4d64a549306c2abd1f0851a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:22:30.283175 19688 log.cc:1196] T 8ae7e409bca743258e77a509bfc7a229 P f4b98f4bd4d64a549306c2abd1f0851a: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-0/wal/wals/8ae7e409bca743258e77a509bfc7a229
I20260807 06:22:30.283396 19625 tablet_bootstrap.cc:492] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf: Bootstrap starting.
I20260807 06:22:30.283514 19688 ts_tablet_manager.cc:1939] T 8ae7e409bca743258e77a509bfc7a229 P f4b98f4bd4d64a549306c2abd1f0851a: Deleting consensus metadata
I20260807 06:22:30.284468 19625 tablet_bootstrap.cc:654] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:30.285200 19623 tablet_bootstrap.cc:492] T 21dd9d873c0242d4aa3c3471a5edab0c P 1e1ca1f8f92046da85f5100209b3209a: Bootstrap starting.
I20260807 06:22:30.285769 19625 tablet_bootstrap.cc:492] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf: No bootstrap required, opened a new log
I20260807 06:22:30.285866 19625 ts_tablet_manager.cc:1397] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260807 06:22:30.286311 19623 tablet_bootstrap.cc:654] T 21dd9d873c0242d4aa3c3471a5edab0c P 1e1ca1f8f92046da85f5100209b3209a: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:30.286345 19625 raft_consensus.cc:348] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:30.286473 19625 raft_consensus.cc:374] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:30.286512 19625 raft_consensus.cc:729] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2c6d77d1d7c4b1888a2d1385067cedf, State: Initialized, Role: FOLLOWER
I20260807 06:22:30.286691 19625 consensus_queue.cc:260] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:30.287215 18857 catalog_manager.cc:4841] TS f4b98f4bd4d64a549306c2abd1f0851a (127.15.254.1:33979): tablet 8ae7e409bca743258e77a509bfc7a229 (table test-table [id=bbbfb7e68a8c4162beb4f4b0524b3ee5]) successfully deleted
I20260807 06:22:30.287554 19625 ts_tablet_manager.cc:1428] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260807 06:22:30.288195 19100 tablet_service.cc:1467] Processing CreateTablet for tablet 21dd9d873c0242d4aa3c3471a5edab0c (DEFAULT_TABLE table=test-table [id=b2321333edbf4342a256114de7a243b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:30.288465 19100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21dd9d873c0242d4aa3c3471a5edab0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:30.289808 19623 tablet_bootstrap.cc:492] T 21dd9d873c0242d4aa3c3471a5edab0c P 1e1ca1f8f92046da85f5100209b3209a: No bootstrap required, opened a new log
I20260807 06:22:30.289899 19623 ts_tablet_manager.cc:1397] T 21dd9d873c0242d4aa3c3471a5edab0c P 1e1ca1f8f92046da85f5100209b3209a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260807 06:22:30.290361 19623 raft_consensus.cc:348] T 21dd9d873c0242d4aa3c3471a5edab0c P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:30.290464 19623 raft_consensus.cc:374] T 21dd9d873c0242d4aa3c3471a5edab0c P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:30.290495 19623 raft_consensus.cc:729] T 21dd9d873c0242d4aa3c3471a5edab0c P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e1ca1f8f92046da85f5100209b3209a, State: Initialized, Role: FOLLOWER
I20260807 06:22:30.290613 19623 consensus_queue.cc:260] T 21dd9d873c0242d4aa3c3471a5edab0c P 1e1ca1f8f92046da85f5100209b3209a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:30.290876 19623 ts_tablet_manager.cc:1428] T 21dd9d873c0242d4aa3c3471a5edab0c P 1e1ca1f8f92046da85f5100209b3209a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 06:22:30.290908 19624 tablet_bootstrap.cc:492] T 21dd9d873c0242d4aa3c3471a5edab0c P f4b98f4bd4d64a549306c2abd1f0851a: Bootstrap starting.
I20260807 06:22:30.291888 19624 tablet_bootstrap.cc:654] T 21dd9d873c0242d4aa3c3471a5edab0c P f4b98f4bd4d64a549306c2abd1f0851a: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:30.280761 19686 ts_tablet_manager.cc:1918] T 8ae7e409bca743258e77a509bfc7a229 P 1e1ca1f8f92046da85f5100209b3209a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:22:30.292224 19686 log.cc:1196] T 8ae7e409bca743258e77a509bfc7a229 P 1e1ca1f8f92046da85f5100209b3209a: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/8ae7e409bca743258e77a509bfc7a229
I20260807 06:22:30.292733 19686 ts_tablet_manager.cc:1939] T 8ae7e409bca743258e77a509bfc7a229 P 1e1ca1f8f92046da85f5100209b3209a: Deleting consensus metadata
I20260807 06:22:30.292719 18858 catalog_manager.cc:4841] TS c2c6d77d1d7c4b1888a2d1385067cedf (127.15.254.2:39045): tablet 8ae7e409bca743258e77a509bfc7a229 (table test-table [id=bbbfb7e68a8c4162beb4f4b0524b3ee5]) successfully deleted
I20260807 06:22:30.294337 18859 catalog_manager.cc:4841] TS 1e1ca1f8f92046da85f5100209b3209a (127.15.254.3:33815): tablet 8ae7e409bca743258e77a509bfc7a229 (table test-table [id=bbbfb7e68a8c4162beb4f4b0524b3ee5]) successfully deleted
I20260807 06:22:30.296824 19624 tablet_bootstrap.cc:492] T 21dd9d873c0242d4aa3c3471a5edab0c P f4b98f4bd4d64a549306c2abd1f0851a: No bootstrap required, opened a new log
I20260807 06:22:30.296944 19624 ts_tablet_manager.cc:1397] T 21dd9d873c0242d4aa3c3471a5edab0c P f4b98f4bd4d64a549306c2abd1f0851a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20260807 06:22:30.297449 19624 raft_consensus.cc:348] T 21dd9d873c0242d4aa3c3471a5edab0c P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:30.297567 19624 raft_consensus.cc:374] T 21dd9d873c0242d4aa3c3471a5edab0c P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:30.297600 19624 raft_consensus.cc:729] T 21dd9d873c0242d4aa3c3471a5edab0c P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b98f4bd4d64a549306c2abd1f0851a, State: Initialized, Role: FOLLOWER
I20260807 06:22:30.297741 19624 consensus_queue.cc:260] T 21dd9d873c0242d4aa3c3471a5edab0c P f4b98f4bd4d64a549306c2abd1f0851a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:30.298017 19624 ts_tablet_manager.cc:1428] T 21dd9d873c0242d4aa3c3471a5edab0c P f4b98f4bd4d64a549306c2abd1f0851a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 06:22:30.385416 19636 raft_consensus.cc:482] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:22:30.385581 19636 raft_consensus.cc:504] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:30.386114 19636 leader_election.cc:288] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4b98f4bd4d64a549306c2abd1f0851a (127.15.254.1:33979), 1e1ca1f8f92046da85f5100209b3209a (127.15.254.3:33815)
I20260807 06:22:30.386654 19127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dd9d873c0242d4aa3c3471a5edab0c" candidate_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" is_pre_election: true
I20260807 06:22:30.386664 19532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dd9d873c0242d4aa3c3471a5edab0c" candidate_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e1ca1f8f92046da85f5100209b3209a" is_pre_election: true
I20260807 06:22:30.386837 19127 raft_consensus.cc:2454] T 21dd9d873c0242d4aa3c3471a5edab0c P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2c6d77d1d7c4b1888a2d1385067cedf in term 0.
I20260807 06:22:30.386871 19532 raft_consensus.cc:2454] T 21dd9d873c0242d4aa3c3471a5edab0c P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2c6d77d1d7c4b1888a2d1385067cedf in term 0.
I20260807 06:22:30.387907 19238 leader_election.cc:302] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2c6d77d1d7c4b1888a2d1385067cedf, f4b98f4bd4d64a549306c2abd1f0851a; no voters:
I20260807 06:22:30.388198 19636 raft_consensus.cc:2781] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:22:30.388267 19636 raft_consensus.cc:482] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:22:30.388315 19636 raft_consensus.cc:3037] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:30.389391 19636 raft_consensus.cc:504] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:30.389716 19636 leader_election.cc:288] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf [CANDIDATE]: Term 1 election: Requested vote from peers f4b98f4bd4d64a549306c2abd1f0851a (127.15.254.1:33979), 1e1ca1f8f92046da85f5100209b3209a (127.15.254.3:33815)
I20260807 06:22:30.390074 19127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dd9d873c0242d4aa3c3471a5edab0c" candidate_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b98f4bd4d64a549306c2abd1f0851a"
I20260807 06:22:30.390110 19532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dd9d873c0242d4aa3c3471a5edab0c" candidate_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e1ca1f8f92046da85f5100209b3209a"
I20260807 06:22:30.390218 19127 raft_consensus.cc:3037] T 21dd9d873c0242d4aa3c3471a5edab0c P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:30.390231 19532 raft_consensus.cc:3037] T 21dd9d873c0242d4aa3c3471a5edab0c P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:30.391227 19127 raft_consensus.cc:2454] T 21dd9d873c0242d4aa3c3471a5edab0c P f4b98f4bd4d64a549306c2abd1f0851a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2c6d77d1d7c4b1888a2d1385067cedf in term 1.
I20260807 06:22:30.391535 19238 leader_election.cc:302] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2c6d77d1d7c4b1888a2d1385067cedf, f4b98f4bd4d64a549306c2abd1f0851a; no voters:
I20260807 06:22:30.391536 19532 raft_consensus.cc:2454] T 21dd9d873c0242d4aa3c3471a5edab0c P 1e1ca1f8f92046da85f5100209b3209a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2c6d77d1d7c4b1888a2d1385067cedf in term 1.
I20260807 06:22:30.391731 19636 raft_consensus.cc:2781] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:30.391848 19636 raft_consensus.cc:686] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf [term 1 LEADER]: Becoming Leader. State: Replica: c2c6d77d1d7c4b1888a2d1385067cedf, State: Running, Role: LEADER
I20260807 06:22:30.391987 19636 consensus_queue.cc:237] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:30.393810 18882 catalog_manager.cc:5495] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf reported cstate change: term changed from 0 to 1, leader changed from <none> to c2c6d77d1d7c4b1888a2d1385067cedf (127.15.254.2). New cstate: current_term: 1 leader_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } health_report { overall_health: UNKNOWN } } }
I20260807 06:22:30.430174 19532 raft_consensus.cc:1261] T 21dd9d873c0242d4aa3c3471a5edab0c P 1e1ca1f8f92046da85f5100209b3209a [term 1 FOLLOWER]: Refusing update from remote peer c2c6d77d1d7c4b1888a2d1385067cedf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:22:30.430298 19127 raft_consensus.cc:1261] T 21dd9d873c0242d4aa3c3471a5edab0c P f4b98f4bd4d64a549306c2abd1f0851a [term 1 FOLLOWER]: Refusing update from remote peer c2c6d77d1d7c4b1888a2d1385067cedf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:22:30.430718 19636 consensus_queue.cc:1035] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:22:30.430894 19638 consensus_queue.cc:1035] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:22:30.438256 19663 mvcc.cc:205] Tried to move back new op lower bound from 7315799041756262400 to 7315799041606963200. Current Snapshot: MvccSnapshot[applied={T|T < 7315799041756262400}]
W20260807 06:22:30.451526 19665 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260807 06:22:32.007041 18882 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:22:32.007229 18882 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:22:32.009843 18882 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce: Sending DeleteTablet for 3 replicas of tablet 21dd9d873c0242d4aa3c3471a5edab0c
I20260807 06:22:32.010571 19315 tablet_service.cc:1514] Processing DeleteTablet for tablet 21dd9d873c0242d4aa3c3471a5edab0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:32 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:33956
I20260807 06:22:32.010655 19506 tablet_service.cc:1514] Processing DeleteTablet for tablet 21dd9d873c0242d4aa3c3471a5edab0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:32 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:59112
I20260807 06:22:32.010677 19100 tablet_service.cc:1514] Processing DeleteTablet for tablet 21dd9d873c0242d4aa3c3471a5edab0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:32 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:50370
I20260807 06:22:32.011185 19732 tablet_replica.cc:330] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf: stopping tablet replica
I20260807 06:22:32.011207 19736 tablet_replica.cc:330] T 21dd9d873c0242d4aa3c3471a5edab0c P f4b98f4bd4d64a549306c2abd1f0851a: stopping tablet replica
I20260807 06:22:32.011335 19736 raft_consensus.cc:2229] T 21dd9d873c0242d4aa3c3471a5edab0c P f4b98f4bd4d64a549306c2abd1f0851a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:22:32.011335 19732 raft_consensus.cc:2229] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:22:32.011459 19736 raft_consensus.cc:2258] T 21dd9d873c0242d4aa3c3471a5edab0c P f4b98f4bd4d64a549306c2abd1f0851a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:32.011555 19732 raft_consensus.cc:2258] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:32.011592 19735 tablet_replica.cc:330] T 21dd9d873c0242d4aa3c3471a5edab0c P 1e1ca1f8f92046da85f5100209b3209a: stopping tablet replica
I20260807 06:22:32.011744 19735 raft_consensus.cc:2229] T 21dd9d873c0242d4aa3c3471a5edab0c P 1e1ca1f8f92046da85f5100209b3209a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:22:32.011862 19735 raft_consensus.cc:2258] T 21dd9d873c0242d4aa3c3471a5edab0c P 1e1ca1f8f92046da85f5100209b3209a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:32.012825 19736 ts_tablet_manager.cc:1905] T 21dd9d873c0242d4aa3c3471a5edab0c P f4b98f4bd4d64a549306c2abd1f0851a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:22:32.014276 19732 ts_tablet_manager.cc:1905] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:22:32.014444 19735 ts_tablet_manager.cc:1905] T 21dd9d873c0242d4aa3c3471a5edab0c P 1e1ca1f8f92046da85f5100209b3209a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:22:32.016129 19736 ts_tablet_manager.cc:1918] T 21dd9d873c0242d4aa3c3471a5edab0c P f4b98f4bd4d64a549306c2abd1f0851a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 06:22:32.016225 19736 log.cc:1196] T 21dd9d873c0242d4aa3c3471a5edab0c P f4b98f4bd4d64a549306c2abd1f0851a: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-0/wal/wals/21dd9d873c0242d4aa3c3471a5edab0c
I20260807 06:22:32.016561 19736 ts_tablet_manager.cc:1939] T 21dd9d873c0242d4aa3c3471a5edab0c P f4b98f4bd4d64a549306c2abd1f0851a: Deleting consensus metadata
I20260807 06:22:32.017158 19732 ts_tablet_manager.cc:1918] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 06:22:32.017253 19732 log.cc:1196] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/21dd9d873c0242d4aa3c3471a5edab0c
I20260807 06:22:32.017270 18857 catalog_manager.cc:4841] TS f4b98f4bd4d64a549306c2abd1f0851a (127.15.254.1:33979): tablet 21dd9d873c0242d4aa3c3471a5edab0c (table test-table [id=b2321333edbf4342a256114de7a243b3]) successfully deleted
I20260807 06:22:32.017503 19735 ts_tablet_manager.cc:1918] T 21dd9d873c0242d4aa3c3471a5edab0c P 1e1ca1f8f92046da85f5100209b3209a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 06:22:32.017567 19732 ts_tablet_manager.cc:1939] T 21dd9d873c0242d4aa3c3471a5edab0c P c2c6d77d1d7c4b1888a2d1385067cedf: Deleting consensus metadata
I20260807 06:22:32.017648 19735 log.cc:1196] T 21dd9d873c0242d4aa3c3471a5edab0c P 1e1ca1f8f92046da85f5100209b3209a: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/21dd9d873c0242d4aa3c3471a5edab0c
I20260807 06:22:32.017939 19735 ts_tablet_manager.cc:1939] T 21dd9d873c0242d4aa3c3471a5edab0c P 1e1ca1f8f92046da85f5100209b3209a: Deleting consensus metadata
I20260807 06:22:32.018428 18858 catalog_manager.cc:4841] TS c2c6d77d1d7c4b1888a2d1385067cedf (127.15.254.2:39045): tablet 21dd9d873c0242d4aa3c3471a5edab0c (table test-table [id=b2321333edbf4342a256114de7a243b3]) successfully deleted
I20260807 06:22:32.018784 18859 catalog_manager.cc:4841] TS 1e1ca1f8f92046da85f5100209b3209a (127.15.254.3:33815): tablet 21dd9d873c0242d4aa3c3471a5edab0c (table test-table [id=b2321333edbf4342a256114de7a243b3]) successfully deleted
Aug 07 06:22:32 dist-test-slave-44d2 krb5kdc[18808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083752, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 06:22:32 dist-test-slave-44d2 krb5kdc[18808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083752, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:32.046780 19744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:32.036577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59306 (local address 127.15.254.62:44813)
0807 06:22:32.036946 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:32.036955 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:32.036970 (+ 15us) server_negotiation.cc:407] Connection header received
0807 06:22:32.037033 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:32.037035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:32.037083 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:32.037178 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:32.037975 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:32.038585 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:32.039666 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:32.039981 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:32.040078 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:32.043056 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:32.043059 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:32.043061 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:32.043089 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:32.045065 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:32.045465 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:32.045468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:32.045470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:32.045560 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:32.045944 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:32.045946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:32.045947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:32.046147 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:32.046263 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:32.046516 (+ 253us) server_negotiation.cc:299] Negotiation successful
0807 06:22:32.046634 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":133,"threads_started":1}
I20260807 06:22:32.049933 18882 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:22:32.050446 18882 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:22:32.056277 19100 tablet_service.cc:1467] Processing CreateTablet for tablet e1d50aacb5a94225831d5e2d74a40148 (DEFAULT_TABLE table=test-table [id=ccdf4c2a355b4811b251f962dae04ddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:32.056285 19315 tablet_service.cc:1467] Processing CreateTablet for tablet e1d50aacb5a94225831d5e2d74a40148 (DEFAULT_TABLE table=test-table [id=ccdf4c2a355b4811b251f962dae04ddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:32.056681 19100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1d50aacb5a94225831d5e2d74a40148. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:32.056743 19315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1d50aacb5a94225831d5e2d74a40148. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:32.056759 19506 tablet_service.cc:1467] Processing CreateTablet for tablet e1d50aacb5a94225831d5e2d74a40148 (DEFAULT_TABLE table=test-table [id=ccdf4c2a355b4811b251f962dae04ddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:32.057164 19506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1d50aacb5a94225831d5e2d74a40148. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:32.059502 19747 tablet_bootstrap.cc:492] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a: Bootstrap starting.
I20260807 06:22:32.060271 19749 tablet_bootstrap.cc:492] T e1d50aacb5a94225831d5e2d74a40148 P c2c6d77d1d7c4b1888a2d1385067cedf: Bootstrap starting.
I20260807 06:22:32.060707 19752 tablet_bootstrap.cc:492] T e1d50aacb5a94225831d5e2d74a40148 P 1e1ca1f8f92046da85f5100209b3209a: Bootstrap starting.
I20260807 06:22:32.060890 19747 tablet_bootstrap.cc:654] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:32.061439 19749 tablet_bootstrap.cc:654] T e1d50aacb5a94225831d5e2d74a40148 P c2c6d77d1d7c4b1888a2d1385067cedf: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:32.061856 19752 tablet_bootstrap.cc:654] T e1d50aacb5a94225831d5e2d74a40148 P 1e1ca1f8f92046da85f5100209b3209a: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:32.062727 19747 tablet_bootstrap.cc:492] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a: No bootstrap required, opened a new log
I20260807 06:22:32.062839 19747 ts_tablet_manager.cc:1397] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260807 06:22:32.063054 19749 tablet_bootstrap.cc:492] T e1d50aacb5a94225831d5e2d74a40148 P c2c6d77d1d7c4b1888a2d1385067cedf: No bootstrap required, opened a new log
I20260807 06:22:32.063169 19749 ts_tablet_manager.cc:1397] T e1d50aacb5a94225831d5e2d74a40148 P c2c6d77d1d7c4b1888a2d1385067cedf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 06:22:32.063432 19747 raft_consensus.cc:348] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:32.063509 19752 tablet_bootstrap.cc:492] T e1d50aacb5a94225831d5e2d74a40148 P 1e1ca1f8f92046da85f5100209b3209a: No bootstrap required, opened a new log
I20260807 06:22:32.063570 19747 raft_consensus.cc:374] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:32.063607 19752 ts_tablet_manager.cc:1397] T e1d50aacb5a94225831d5e2d74a40148 P 1e1ca1f8f92046da85f5100209b3209a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 06:22:32.063617 19747 raft_consensus.cc:729] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b98f4bd4d64a549306c2abd1f0851a, State: Initialized, Role: FOLLOWER
I20260807 06:22:32.063679 19749 raft_consensus.cc:348] T e1d50aacb5a94225831d5e2d74a40148 P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:32.063817 19749 raft_consensus.cc:374] T e1d50aacb5a94225831d5e2d74a40148 P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:32.063774 19747 consensus_queue.cc:260] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:32.063865 19749 raft_consensus.cc:729] T e1d50aacb5a94225831d5e2d74a40148 P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2c6d77d1d7c4b1888a2d1385067cedf, State: Initialized, Role: FOLLOWER
I20260807 06:22:32.064033 19749 consensus_queue.cc:260] T e1d50aacb5a94225831d5e2d74a40148 P c2c6d77d1d7c4b1888a2d1385067cedf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:32.064281 19747 ts_tablet_manager.cc:1428] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 06:22:32.064560 19749 ts_tablet_manager.cc:1428] T e1d50aacb5a94225831d5e2d74a40148 P c2c6d77d1d7c4b1888a2d1385067cedf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 06:22:32.064231 19752 raft_consensus.cc:348] T e1d50aacb5a94225831d5e2d74a40148 P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:32.064971 19752 raft_consensus.cc:374] T e1d50aacb5a94225831d5e2d74a40148 P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:32.065042 19752 raft_consensus.cc:729] T e1d50aacb5a94225831d5e2d74a40148 P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e1ca1f8f92046da85f5100209b3209a, State: Initialized, Role: FOLLOWER
I20260807 06:22:32.065356 19752 consensus_queue.cc:260] T e1d50aacb5a94225831d5e2d74a40148 P 1e1ca1f8f92046da85f5100209b3209a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:32.065924 19752 ts_tablet_manager.cc:1428] T e1d50aacb5a94225831d5e2d74a40148 P 1e1ca1f8f92046da85f5100209b3209a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 06:22:32.115212 19759 raft_consensus.cc:482] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:22:32.115396 19759 raft_consensus.cc:504] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:32.116672 19759 leader_election.cc:288] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2c6d77d1d7c4b1888a2d1385067cedf (127.15.254.2:39045), 1e1ca1f8f92046da85f5100209b3209a (127.15.254.3:33815)
I20260807 06:22:32.121825 19764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:32.117103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:37597 (local address 127.15.254.3:33815)
0807 06:22:32.117821 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:32.117833 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:32.117849 (+ 16us) server_negotiation.cc:407] Connection header received
0807 06:22:32.117907 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:32.117909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:32.117999 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:32.118093 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:32.118347 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:32.119127 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:32.120586 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:32.121230 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:32.121341 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:32.121396 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:32.121603 (+ 207us) server_negotiation.cc:299] Negotiation successful
0807 06:22:32.121685 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":111,"threads_started":1}
I20260807 06:22:32.121825 19763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:32.116989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:52663 (local address 127.15.254.2:39045)
0807 06:22:32.117325 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:32.117335 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:32.117421 (+ 86us) server_negotiation.cc:407] Connection header received
0807 06:22:32.117582 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:32.117586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:32.117645 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:32.117740 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:32.118007 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:32.118828 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:32.120425 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:32.121120 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:32.121253 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:32.121329 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:32.121590 (+ 261us) server_negotiation.cc:299] Negotiation successful
0807 06:22:32.121679 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":109,"threads_started":1}
I20260807 06:22:32.121850 19762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:32.116911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:33815 (local address 127.15.254.1:37597)
0807 06:22:32.117432 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:32.117448 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:32.117585 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:32.118116 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:32.118118 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:32.118133 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:32.118229 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:32.118233 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:32.119266 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:32.119269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:32.120414 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:32.120426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:32.121340 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:32.121345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:32.121459 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:32.121479 (+ 20us) client_negotiation.cc:769] Sending connection context
0807 06:22:32.121549 (+ 70us) client_negotiation.cc:240] Negotiation successful
0807 06:22:32.121650 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":44,"thread_start_us":121,"threads_started":1}
I20260807 06:22:32.121850 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:32.116885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:39045 (local address 127.15.254.1:52663)
0807 06:22:32.117324 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:32.117338 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:32.117461 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:32.117763 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:32.117765 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:32.117780 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:32.117886 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:32.117890 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:32.118951 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:32.118955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:32.120293 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:32.120305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:32.121252 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:32.121255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:32.121381 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:32.121452 (+ 71us) client_negotiation.cc:769] Sending connection context
0807 06:22:32.121539 (+ 87us) client_negotiation.cc:240] Negotiation successful
0807 06:22:32.121647 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":21,"thread_start_us":148,"threads_started":1}
I20260807 06:22:32.122411 19532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1d50aacb5a94225831d5e2d74a40148" candidate_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e1ca1f8f92046da85f5100209b3209a" is_pre_election: true
I20260807 06:22:32.122599 19532 raft_consensus.cc:2454] T e1d50aacb5a94225831d5e2d74a40148 P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4b98f4bd4d64a549306c2abd1f0851a in term 0.
I20260807 06:22:32.122700 19344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1d50aacb5a94225831d5e2d74a40148" candidate_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" is_pre_election: true
I20260807 06:22:32.122923 19344 raft_consensus.cc:2454] T e1d50aacb5a94225831d5e2d74a40148 P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4b98f4bd4d64a549306c2abd1f0851a in term 0.
I20260807 06:22:32.123124 19030 leader_election.cc:302] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e1ca1f8f92046da85f5100209b3209a, f4b98f4bd4d64a549306c2abd1f0851a; no voters:
I20260807 06:22:32.123409 19759 raft_consensus.cc:2781] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:22:32.123469 19759 raft_consensus.cc:482] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:22:32.123498 19759 raft_consensus.cc:3037] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:32.124668 19759 raft_consensus.cc:504] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:32.125084 19759 leader_election.cc:288] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a [CANDIDATE]: Term 1 election: Requested vote from peers c2c6d77d1d7c4b1888a2d1385067cedf (127.15.254.2:39045), 1e1ca1f8f92046da85f5100209b3209a (127.15.254.3:33815)
I20260807 06:22:32.125456 19344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1d50aacb5a94225831d5e2d74a40148" candidate_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf"
I20260807 06:22:32.125617 19344 raft_consensus.cc:3037] T e1d50aacb5a94225831d5e2d74a40148 P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:32.125552 19532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1d50aacb5a94225831d5e2d74a40148" candidate_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e1ca1f8f92046da85f5100209b3209a"
I20260807 06:22:32.125742 19532 raft_consensus.cc:3037] T e1d50aacb5a94225831d5e2d74a40148 P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:32.126628 19344 raft_consensus.cc:2454] T e1d50aacb5a94225831d5e2d74a40148 P c2c6d77d1d7c4b1888a2d1385067cedf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4b98f4bd4d64a549306c2abd1f0851a in term 1.
I20260807 06:22:32.126751 19532 raft_consensus.cc:2454] T e1d50aacb5a94225831d5e2d74a40148 P 1e1ca1f8f92046da85f5100209b3209a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4b98f4bd4d64a549306c2abd1f0851a in term 1.
I20260807 06:22:32.126942 19028 leader_election.cc:302] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2c6d77d1d7c4b1888a2d1385067cedf, f4b98f4bd4d64a549306c2abd1f0851a; no voters:
I20260807 06:22:32.127133 19759 raft_consensus.cc:2781] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:32.127377 19759 raft_consensus.cc:686] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a [term 1 LEADER]: Becoming Leader. State: Replica: f4b98f4bd4d64a549306c2abd1f0851a, State: Running, Role: LEADER
I20260807 06:22:32.127653 19759 consensus_queue.cc:237] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:32.130324 18879 catalog_manager.cc:5495] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a reported cstate change: term changed from 0 to 1, leader changed from <none> to f4b98f4bd4d64a549306c2abd1f0851a (127.15.254.1). New cstate: current_term: 1 leader_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } health_report { overall_health: UNKNOWN } } }
I20260807 06:22:32.148638 19773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:32.144611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37458 (local address 127.15.254.1:33979)
0807 06:22:32.144896 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:32.144903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:32.144914 (+ 11us) server_negotiation.cc:407] Connection header received
0807 06:22:32.144988 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:32.144990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:32.145066 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:32.145178 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:22:32.145415 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:32.145946 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:32.146759 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:32.146994 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:32.147095 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:32.147387 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:22:32.148117 (+ 730us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:22:32.148236 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:32.148299 (+ 63us) server_negotiation.cc:299] Negotiation successful
0807 06:22:32.148363 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":134,"threads_started":1}
I20260807 06:22:32.159224 19764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:32.155979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:43757 (local address 127.15.254.3:33815)
0807 06:22:32.156111 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:32.156115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:32.156320 (+ 205us) server_negotiation.cc:407] Connection header received
0807 06:22:32.156418 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:32.156420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:32.156465 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:32.156589 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:22:32.156783 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:32.157299 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:32.158137 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:32.158379 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:32.158480 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:32.158749 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:22:32.158884 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:22:32.158953 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:32.159043 (+ 90us) server_negotiation.cc:299] Negotiation successful
0807 06:22:32.159107 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 06:22:32.159243 19776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:32.155906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:33815 (local address 127.15.254.1:43757)
0807 06:22:32.156247 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:32.156259 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:32.156347 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:32.156598 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:32.156600 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:32.156606 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 06:22:32.156709 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:32.156713 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:32.157399 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:32.157401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:32.157993 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:32.158000 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:32.158500 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:32.158502 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:32.158575 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:32.158619 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 06:22:32.158975 (+ 356us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 06:22:32.158981 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 06:22:32.159031 (+ 50us) client_negotiation.cc:240] Negotiation successful
0807 06:22:32.159115 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":110,"threads_started":1}
I20260807 06:22:32.165848 19344 raft_consensus.cc:1261] T e1d50aacb5a94225831d5e2d74a40148 P c2c6d77d1d7c4b1888a2d1385067cedf [term 1 FOLLOWER]: Refusing update from remote peer f4b98f4bd4d64a549306c2abd1f0851a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:22:32.165899 19532 raft_consensus.cc:1261] T e1d50aacb5a94225831d5e2d74a40148 P 1e1ca1f8f92046da85f5100209b3209a [term 1 FOLLOWER]: Refusing update from remote peer f4b98f4bd4d64a549306c2abd1f0851a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:22:32.166347 19768 consensus_queue.cc:1035] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:22:32.166518 19759 consensus_queue.cc:1035] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:22:32.170176 19707 mvcc.cc:205] Tried to move back new op lower bound from 7315799048866897920 to 7315799048717332480. Current Snapshot: MvccSnapshot[applied={T|T < 7315799048866897920}]
I20260807 06:22:32.186228 19773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:32.182525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:57095 (local address 127.15.254.1:33979)
0807 06:22:32.182634 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:32.182640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:32.182852 (+ 212us) server_negotiation.cc:407] Connection header received
0807 06:22:32.182997 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:32.183000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:32.183051 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:32.183132 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:22:32.183403 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:32.184030 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:32.184976 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:32.185284 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:32.185408 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:32.185691 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:22:32.185783 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:22:32.185856 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:32.186014 (+ 158us) server_negotiation.cc:299] Negotiation successful
0807 06:22:32.186100 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260807 06:22:32.186209 19786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:32.182474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:33979 (local address 127.15.254.3:57095)
0807 06:22:32.182768 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:32.182781 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:32.182897 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:32.183146 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:32.183148 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:32.183158 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 06:22:32.183276 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:32.183283 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:32.184185 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:32.184188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:32.184872 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:32.184882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:32.185413 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:32.185417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:32.185514 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:32.185542 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 06:22:32.185882 (+ 340us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 06:22:32.185890 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 06:22:32.185982 (+ 92us) client_negotiation.cc:240] Negotiation successful
0807 06:22:32.186081 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":125,"threads_started":1}
I20260807 06:22:32.210318 18879 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:22:32.210503 18879 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:22:32.212829 18879 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce: Sending DeleteTablet for 3 replicas of tablet e1d50aacb5a94225831d5e2d74a40148
I20260807 06:22:32.213423 19100 tablet_service.cc:1514] Processing DeleteTablet for tablet e1d50aacb5a94225831d5e2d74a40148 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:32 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:50370
I20260807 06:22:32.213408 19315 tablet_service.cc:1514] Processing DeleteTablet for tablet e1d50aacb5a94225831d5e2d74a40148 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:32 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:33956
I20260807 06:22:32.213598 19736 tablet_replica.cc:330] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a: stopping tablet replica
I20260807 06:22:32.213617 19732 tablet_replica.cc:330] T e1d50aacb5a94225831d5e2d74a40148 P c2c6d77d1d7c4b1888a2d1385067cedf: stopping tablet replica
I20260807 06:22:32.213636 19506 tablet_service.cc:1514] Processing DeleteTablet for tablet e1d50aacb5a94225831d5e2d74a40148 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:32 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:59112
I20260807 06:22:32.213706 19736 raft_consensus.cc:2229] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:22:32.213711 19732 raft_consensus.cc:2229] T e1d50aacb5a94225831d5e2d74a40148 P c2c6d77d1d7c4b1888a2d1385067cedf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:22:32.213814 19732 raft_consensus.cc:2258] T e1d50aacb5a94225831d5e2d74a40148 P c2c6d77d1d7c4b1888a2d1385067cedf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:32.213830 19735 tablet_replica.cc:330] T e1d50aacb5a94225831d5e2d74a40148 P 1e1ca1f8f92046da85f5100209b3209a: stopping tablet replica
I20260807 06:22:32.213945 19735 raft_consensus.cc:2229] T e1d50aacb5a94225831d5e2d74a40148 P 1e1ca1f8f92046da85f5100209b3209a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:22:32.213999 19736 raft_consensus.cc:2258] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:32.214052 19735 raft_consensus.cc:2258] T e1d50aacb5a94225831d5e2d74a40148 P 1e1ca1f8f92046da85f5100209b3209a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:32.214669 19736 ts_tablet_manager.cc:1905] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:22:32.214905 19735 ts_tablet_manager.cc:1905] T e1d50aacb5a94225831d5e2d74a40148 P 1e1ca1f8f92046da85f5100209b3209a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:22:32.215108 18879 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 06:22:32.215227 19732 ts_tablet_manager.cc:1905] T e1d50aacb5a94225831d5e2d74a40148 P c2c6d77d1d7c4b1888a2d1385067cedf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260807 06:22:32.215509 18879 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:22:32.217468 19735 ts_tablet_manager.cc:1918] T e1d50aacb5a94225831d5e2d74a40148 P 1e1ca1f8f92046da85f5100209b3209a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:22:32.217531 19735 log.cc:1196] T e1d50aacb5a94225831d5e2d74a40148 P 1e1ca1f8f92046da85f5100209b3209a: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/e1d50aacb5a94225831d5e2d74a40148
I20260807 06:22:32.217519 19736 ts_tablet_manager.cc:1918] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:22:32.217592 19736 log.cc:1196] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-0/wal/wals/e1d50aacb5a94225831d5e2d74a40148
I20260807 06:22:32.217733 19735 ts_tablet_manager.cc:1939] T e1d50aacb5a94225831d5e2d74a40148 P 1e1ca1f8f92046da85f5100209b3209a: Deleting consensus metadata
I20260807 06:22:32.217864 19736 ts_tablet_manager.cc:1939] T e1d50aacb5a94225831d5e2d74a40148 P f4b98f4bd4d64a549306c2abd1f0851a: Deleting consensus metadata
I20260807 06:22:32.218125 19732 ts_tablet_manager.cc:1918] T e1d50aacb5a94225831d5e2d74a40148 P c2c6d77d1d7c4b1888a2d1385067cedf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:22:32.218194 19732 log.cc:1196] T e1d50aacb5a94225831d5e2d74a40148 P c2c6d77d1d7c4b1888a2d1385067cedf: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/e1d50aacb5a94225831d5e2d74a40148
I20260807 06:22:32.218446 19732 ts_tablet_manager.cc:1939] T e1d50aacb5a94225831d5e2d74a40148 P c2c6d77d1d7c4b1888a2d1385067cedf: Deleting consensus metadata
I20260807 06:22:32.218554 18859 catalog_manager.cc:4841] TS 1e1ca1f8f92046da85f5100209b3209a (127.15.254.3:33815): tablet e1d50aacb5a94225831d5e2d74a40148 (table test-table [id=ccdf4c2a355b4811b251f962dae04ddf]) successfully deleted
I20260807 06:22:32.218616 18857 catalog_manager.cc:4841] TS f4b98f4bd4d64a549306c2abd1f0851a (127.15.254.1:33979): tablet e1d50aacb5a94225831d5e2d74a40148 (table test-table [id=ccdf4c2a355b4811b251f962dae04ddf]) successfully deleted
I20260807 06:22:32.219254 18858 catalog_manager.cc:4841] TS c2c6d77d1d7c4b1888a2d1385067cedf (127.15.254.2:39045): tablet e1d50aacb5a94225831d5e2d74a40148 (table test-table [id=ccdf4c2a355b4811b251f962dae04ddf]) successfully deleted
I20260807 06:22:32.222895 19100 tablet_service.cc:1467] Processing CreateTablet for tablet cf3764d1fa0541368218b5c90d35f1e0 (DEFAULT_TABLE table=test-table [id=c109d14efd6440079c5fe6a818ac0bbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:32.223125 19315 tablet_service.cc:1467] Processing CreateTablet for tablet cf3764d1fa0541368218b5c90d35f1e0 (DEFAULT_TABLE table=test-table [id=c109d14efd6440079c5fe6a818ac0bbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:32.223212 19100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf3764d1fa0541368218b5c90d35f1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:32.223379 19315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf3764d1fa0541368218b5c90d35f1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:32.225636 19749 tablet_bootstrap.cc:492] T cf3764d1fa0541368218b5c90d35f1e0 P c2c6d77d1d7c4b1888a2d1385067cedf: Bootstrap starting.
I20260807 06:22:32.226380 19749 tablet_bootstrap.cc:654] T cf3764d1fa0541368218b5c90d35f1e0 P c2c6d77d1d7c4b1888a2d1385067cedf: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:32.227195 19747 tablet_bootstrap.cc:492] T cf3764d1fa0541368218b5c90d35f1e0 P f4b98f4bd4d64a549306c2abd1f0851a: Bootstrap starting.
I20260807 06:22:32.227362 19506 tablet_service.cc:1467] Processing CreateTablet for tablet cf3764d1fa0541368218b5c90d35f1e0 (DEFAULT_TABLE table=test-table [id=c109d14efd6440079c5fe6a818ac0bbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:32.227696 19506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf3764d1fa0541368218b5c90d35f1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:32.228147 19747 tablet_bootstrap.cc:654] T cf3764d1fa0541368218b5c90d35f1e0 P f4b98f4bd4d64a549306c2abd1f0851a: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:32.229008 19749 tablet_bootstrap.cc:492] T cf3764d1fa0541368218b5c90d35f1e0 P c2c6d77d1d7c4b1888a2d1385067cedf: No bootstrap required, opened a new log
I20260807 06:22:32.229138 19749 ts_tablet_manager.cc:1397] T cf3764d1fa0541368218b5c90d35f1e0 P c2c6d77d1d7c4b1888a2d1385067cedf: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260807 06:22:32.229558 19749 raft_consensus.cc:348] T cf3764d1fa0541368218b5c90d35f1e0 P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:32.229694 19749 raft_consensus.cc:374] T cf3764d1fa0541368218b5c90d35f1e0 P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:32.229740 19749 raft_consensus.cc:729] T cf3764d1fa0541368218b5c90d35f1e0 P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2c6d77d1d7c4b1888a2d1385067cedf, State: Initialized, Role: FOLLOWER
I20260807 06:22:32.229897 19749 consensus_queue.cc:260] T cf3764d1fa0541368218b5c90d35f1e0 P c2c6d77d1d7c4b1888a2d1385067cedf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:32.230190 19749 ts_tablet_manager.cc:1428] T cf3764d1fa0541368218b5c90d35f1e0 P c2c6d77d1d7c4b1888a2d1385067cedf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 06:22:32.230559 19752 tablet_bootstrap.cc:492] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a: Bootstrap starting.
I20260807 06:22:32.231523 19752 tablet_bootstrap.cc:654] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:32.232827 19747 tablet_bootstrap.cc:492] T cf3764d1fa0541368218b5c90d35f1e0 P f4b98f4bd4d64a549306c2abd1f0851a: No bootstrap required, opened a new log
I20260807 06:22:32.232959 19747 ts_tablet_manager.cc:1397] T cf3764d1fa0541368218b5c90d35f1e0 P f4b98f4bd4d64a549306c2abd1f0851a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260807 06:22:32.232999 19752 tablet_bootstrap.cc:492] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a: No bootstrap required, opened a new log
I20260807 06:22:32.233085 19752 ts_tablet_manager.cc:1397] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 06:22:32.233439 19747 raft_consensus.cc:348] T cf3764d1fa0541368218b5c90d35f1e0 P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:32.233568 19747 raft_consensus.cc:374] T cf3764d1fa0541368218b5c90d35f1e0 P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:32.233614 19747 raft_consensus.cc:729] T cf3764d1fa0541368218b5c90d35f1e0 P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b98f4bd4d64a549306c2abd1f0851a, State: Initialized, Role: FOLLOWER
I20260807 06:22:32.233560 19752 raft_consensus.cc:348] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:32.233697 19752 raft_consensus.cc:374] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:32.233738 19752 raft_consensus.cc:729] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e1ca1f8f92046da85f5100209b3209a, State: Initialized, Role: FOLLOWER
I20260807 06:22:32.233757 19747 consensus_queue.cc:260] T cf3764d1fa0541368218b5c90d35f1e0 P f4b98f4bd4d64a549306c2abd1f0851a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:32.233863 19752 consensus_queue.cc:260] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:32.234130 19747 ts_tablet_manager.cc:1428] T cf3764d1fa0541368218b5c90d35f1e0 P f4b98f4bd4d64a549306c2abd1f0851a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 06:22:32.234184 19752 ts_tablet_manager.cc:1428] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 06:22:32.455322 19705 raft_consensus.cc:482] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:22:32.455478 19705 raft_consensus.cc:504] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:32.455921 19705 leader_election.cc:288] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4b98f4bd4d64a549306c2abd1f0851a (127.15.254.1:33979), c2c6d77d1d7c4b1888a2d1385067cedf (127.15.254.2:39045)
I20260807 06:22:32.456501 19344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf3764d1fa0541368218b5c90d35f1e0" candidate_uuid: "1e1ca1f8f92046da85f5100209b3209a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" is_pre_election: true
I20260807 06:22:32.456473 19127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf3764d1fa0541368218b5c90d35f1e0" candidate_uuid: "1e1ca1f8f92046da85f5100209b3209a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" is_pre_election: true
I20260807 06:22:32.456692 19344 raft_consensus.cc:2454] T cf3764d1fa0541368218b5c90d35f1e0 P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e1ca1f8f92046da85f5100209b3209a in term 0.
I20260807 06:22:32.456697 19127 raft_consensus.cc:2454] T cf3764d1fa0541368218b5c90d35f1e0 P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e1ca1f8f92046da85f5100209b3209a in term 0.
I20260807 06:22:32.457043 19442 leader_election.cc:302] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e1ca1f8f92046da85f5100209b3209a, c2c6d77d1d7c4b1888a2d1385067cedf; no voters:
I20260807 06:22:32.457293 19705 raft_consensus.cc:2781] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:22:32.457363 19705 raft_consensus.cc:482] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:22:32.457392 19705 raft_consensus.cc:3037] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:32.458552 19705 raft_consensus.cc:504] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:32.458920 19705 leader_election.cc:288] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a [CANDIDATE]: Term 1 election: Requested vote from peers f4b98f4bd4d64a549306c2abd1f0851a (127.15.254.1:33979), c2c6d77d1d7c4b1888a2d1385067cedf (127.15.254.2:39045)
I20260807 06:22:32.459358 19344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf3764d1fa0541368218b5c90d35f1e0" candidate_uuid: "1e1ca1f8f92046da85f5100209b3209a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf"
I20260807 06:22:32.459363 19127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf3764d1fa0541368218b5c90d35f1e0" candidate_uuid: "1e1ca1f8f92046da85f5100209b3209a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b98f4bd4d64a549306c2abd1f0851a"
I20260807 06:22:32.459517 19344 raft_consensus.cc:3037] T cf3764d1fa0541368218b5c90d35f1e0 P c2c6d77d1d7c4b1888a2d1385067cedf [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:32.459532 19127 raft_consensus.cc:3037] T cf3764d1fa0541368218b5c90d35f1e0 P f4b98f4bd4d64a549306c2abd1f0851a [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:32.460644 19344 raft_consensus.cc:2454] T cf3764d1fa0541368218b5c90d35f1e0 P c2c6d77d1d7c4b1888a2d1385067cedf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e1ca1f8f92046da85f5100209b3209a in term 1.
I20260807 06:22:32.460701 19127 raft_consensus.cc:2454] T cf3764d1fa0541368218b5c90d35f1e0 P f4b98f4bd4d64a549306c2abd1f0851a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e1ca1f8f92046da85f5100209b3209a in term 1.
I20260807 06:22:32.461001 19442 leader_election.cc:302] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e1ca1f8f92046da85f5100209b3209a, c2c6d77d1d7c4b1888a2d1385067cedf; no voters:
I20260807 06:22:32.461213 19705 raft_consensus.cc:2781] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:32.461323 19705 raft_consensus.cc:686] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a [term 1 LEADER]: Becoming Leader. State: Replica: 1e1ca1f8f92046da85f5100209b3209a, State: Running, Role: LEADER
I20260807 06:22:32.461480 19705 consensus_queue.cc:237] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } }
I20260807 06:22:32.463444 18882 catalog_manager.cc:5495] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e1ca1f8f92046da85f5100209b3209a (127.15.254.3). New cstate: current_term: 1 leader_uuid: "1e1ca1f8f92046da85f5100209b3209a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e1ca1f8f92046da85f5100209b3209a" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33815 } health_report { overall_health: HEALTHY } } }
I20260807 06:22:32.483328 19764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:32.479751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47554 (local address 127.15.254.3:33815)
0807 06:22:32.479914 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:32.479919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:32.479935 (+ 16us) server_negotiation.cc:407] Connection header received
0807 06:22:32.480116 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:32.480119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:32.480178 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:32.480267 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:22:32.480611 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:32.481212 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:32.482144 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:32.482406 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:32.482531 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:32.482809 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:22:32.482918 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:22:32.483012 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:32.483118 (+ 106us) server_negotiation.cc:299] Negotiation successful
0807 06:22:32.483170 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 06:22:32.492689 19786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:32.489501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:33815 (local address 127.15.254.3:49753)
0807 06:22:32.489660 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:32.489682 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:32.489772 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:32.490051 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:32.490053 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:32.490060 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 06:22:32.490157 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:32.490168 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:32.490900 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:32.490904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:32.491474 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:32.491482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:32.491924 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:32.491928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:32.492037 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:32.492060 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 06:22:32.492335 (+ 275us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 06:22:32.492343 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 06:22:32.492409 (+ 66us) client_negotiation.cc:240] Negotiation successful
0807 06:22:32.492538 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":26}
I20260807 06:22:32.492689 19764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:32.489714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:49753 (local address 127.15.254.3:33815)
0807 06:22:32.489846 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:32.489851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:32.489863 (+ 12us) server_negotiation.cc:407] Connection header received
0807 06:22:32.489899 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:32.489902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:32.489946 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:32.490041 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:22:32.490266 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:32.490781 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:32.491595 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:32.491817 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:32.491927 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:32.492183 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:22:32.492252 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:22:32.492330 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:32.492418 (+ 88us) server_negotiation.cc:299] Negotiation successful
0807 06:22:32.492508 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":52}
I20260807 06:22:32.498919 19344 raft_consensus.cc:1261] T cf3764d1fa0541368218b5c90d35f1e0 P c2c6d77d1d7c4b1888a2d1385067cedf [term 1 FOLLOWER]: Refusing update from remote peer 1e1ca1f8f92046da85f5100209b3209a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:22:32.498919 19127 raft_consensus.cc:1261] T cf3764d1fa0541368218b5c90d35f1e0 P f4b98f4bd4d64a549306c2abd1f0851a [term 1 FOLLOWER]: Refusing update from remote peer 1e1ca1f8f92046da85f5100209b3209a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:22:32.499362 19722 consensus_queue.cc:1035] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b98f4bd4d64a549306c2abd1f0851a" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:22:32.499544 19777 consensus_queue.cc:1035] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2c6d77d1d7c4b1888a2d1385067cedf" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 06:22:32.514587 19701 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260807 06:22:34.071223 18882 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:22:34.071410 18882 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:22:34.073999 18882 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09fb44b5c342445c84c0a88d46b1d8ce: Sending DeleteTablet for 3 replicas of tablet cf3764d1fa0541368218b5c90d35f1e0
I20260807 06:22:34.074623 19315 tablet_service.cc:1514] Processing DeleteTablet for tablet cf3764d1fa0541368218b5c90d35f1e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:34 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:33956
I20260807 06:22:34.074744 19100 tablet_service.cc:1514] Processing DeleteTablet for tablet cf3764d1fa0541368218b5c90d35f1e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:34 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:50370
I20260807 06:22:34.074772 19506 tablet_service.cc:1514] Processing DeleteTablet for tablet cf3764d1fa0541368218b5c90d35f1e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:34 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:59112
I20260807 06:22:34.075117 19828 tablet_replica.cc:330] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a: stopping tablet replica
I20260807 06:22:34.075238 19828 raft_consensus.cc:2229] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:22:34.075416 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 18990
I20260807 06:22:34.075925 19828 raft_consensus.cc:2258] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:34.079475 19828 ts_tablet_manager.cc:1905] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:22:34.082211 19828 ts_tablet_manager.cc:1918] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 06:22:34.082370 19828 log.cc:1196] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/cf3764d1fa0541368218b5c90d35f1e0
I20260807 06:22:34.082726 19828 ts_tablet_manager.cc:1939] T cf3764d1fa0541368218b5c90d35f1e0 P 1e1ca1f8f92046da85f5100209b3209a: Deleting consensus metadata
I20260807 06:22:34.083532 18859 catalog_manager.cc:4841] TS 1e1ca1f8f92046da85f5100209b3209a (127.15.254.3:33815): tablet cf3764d1fa0541368218b5c90d35f1e0 (table test-table [id=c109d14efd6440079c5fe6a818ac0bbd]) successfully deleted
I20260807 06:22:34.084708 19826 tablet_replica.cc:330] T cf3764d1fa0541368218b5c90d35f1e0 P c2c6d77d1d7c4b1888a2d1385067cedf: stopping tablet replica
I20260807 06:22:34.084882 19826 raft_consensus.cc:2229] T cf3764d1fa0541368218b5c90d35f1e0 P c2c6d77d1d7c4b1888a2d1385067cedf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:22:34.085040 19826 raft_consensus.cc:2258] T cf3764d1fa0541368218b5c90d35f1e0 P c2c6d77d1d7c4b1888a2d1385067cedf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:34.085685 19826 ts_tablet_manager.cc:1905] T cf3764d1fa0541368218b5c90d35f1e0 P c2c6d77d1d7c4b1888a2d1385067cedf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:22:34.087914 19826 ts_tablet_manager.cc:1918] T cf3764d1fa0541368218b5c90d35f1e0 P c2c6d77d1d7c4b1888a2d1385067cedf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 06:22:34.088050 19826 log.cc:1196] T cf3764d1fa0541368218b5c90d35f1e0 P c2c6d77d1d7c4b1888a2d1385067cedf: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/cf3764d1fa0541368218b5c90d35f1e0
I20260807 06:22:34.088318 19826 ts_tablet_manager.cc:1939] T cf3764d1fa0541368218b5c90d35f1e0 P c2c6d77d1d7c4b1888a2d1385067cedf: Deleting consensus metadata
I20260807 06:22:34.089105 18858 catalog_manager.cc:4841] TS c2c6d77d1d7c4b1888a2d1385067cedf (127.15.254.2:39045): tablet cf3764d1fa0541368218b5c90d35f1e0 (table test-table [id=c109d14efd6440079c5fe6a818ac0bbd]) successfully deleted
W20260807 06:22:34.095261 18857 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.254.1:33979 (error 108)
I20260807 06:22:34.095465 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 19216
I20260807 06:22:34.103688 19831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:34.103081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:33979 (local address 127.0.0.1:37464)
0807 06:22:34.103428 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:34.103540 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.1:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":106,"threads_started":1}
W20260807 06:22:34.103890 18857 catalog_manager.cc:4584] TS f4b98f4bd4d64a549306c2abd1f0851a (127.15.254.1:33979): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cf3764d1fa0541368218b5c90d35f1e0: Network error: Client connection negotiation failed: client connection to 127.15.254.1:33979: connect: Connection refused (error 111)
I20260807 06:22:34.111213 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 19417
I20260807 06:22:34.125600 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 18829
2026-08-07T06:22:34Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6629 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:22:34 dist-test-slave-44d2 krb5kdc[19835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:22:34 dist-test-slave-44d2 krb5kdc[19835](info): set up 2 sockets
Aug 07 06:22:34 dist-test-slave-44d2 krb5kdc[19835](info): commencing operation
krb5kdc: starting...
W20260807 06:22:35.349176 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:22:35 dist-test-slave-44d2 krb5kdc[19835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:22:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:22:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:22:35.484251 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:46491
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:38959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:46491
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:35.598941 19858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:35.599210 19858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:35.599248 19858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:35.603503 19858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:22:35.603586 19858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:35.603641 19858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:35.603688 19858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:22:35.603725 19858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:22:35.609205 19858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:38959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:46491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:46491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:35.610217 19858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:35.611184 19858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:35.620289 19868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:35.620352 19864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:35.620440 19865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:35.620527 19858 server_base.cc:1034] running on GCE node
I20260807 06:22:35.621320 19858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:35.622551 19858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:35.623749 19858 hybrid_clock.cc:648] HybridClock initialized: now 1786083755623730 us; error 34 us; skew 500 ppm
Aug 07 06:22:35 dist-test-slave-44d2 krb5kdc[19835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083755, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:35.630883 19858 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:22:35.631866 19858 webserver.cc:460] Webserver started at http://127.15.254.62:44633/ using document root <none> and password file <none>
I20260807 06:22:35.632473 19858 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:35.632640 19858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:35.632850 19858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:35.635190 19858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "4ec53b7046604f9a9294453519132be4"
format_stamp: "Formatted at 2026-08-07 06:22:35 on dist-test-slave-44d2"
server_key: "2ad7492ca1605ed1fe2a376eed5ab745"
server_key_iv: "f710d4e55966a728723d8cffee88db38"
server_key_version: "encryptionkey@0"
I20260807 06:22:35.635798 19858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "4ec53b7046604f9a9294453519132be4"
format_stamp: "Formatted at 2026-08-07 06:22:35 on dist-test-slave-44d2"
server_key: "2ad7492ca1605ed1fe2a376eed5ab745"
server_key_iv: "f710d4e55966a728723d8cffee88db38"
server_key_version: "encryptionkey@0"
I20260807 06:22:35.639981 19858 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 06:22:35.642783 19883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:35.644083 19858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260807 06:22:35.644236 19858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "4ec53b7046604f9a9294453519132be4"
format_stamp: "Formatted at 2026-08-07 06:22:35 on dist-test-slave-44d2"
server_key: "2ad7492ca1605ed1fe2a376eed5ab745"
server_key_iv: "f710d4e55966a728723d8cffee88db38"
server_key_version: "encryptionkey@0"
I20260807 06:22:35.644371 19858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:35.667224 19858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:35.671144 19858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:35.671375 19858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:35.678952 19858 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:46491
I20260807 06:22:35.678982 19953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:46491 every 8 connection(s)
I20260807 06:22:35.679968 19858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:22:35.681475 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 19858
I20260807 06:22:35.681576 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:22:35.681898 16376 external_mini_cluster.cc:1428] Setting key 00fd63068b4a74fbd4001d44c7709d6f
I20260807 06:22:35.683322 19955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:35.689077 19955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ec53b7046604f9a9294453519132be4: Bootstrap starting.
Aug 07 06:22:35 dist-test-slave-44d2 krb5kdc[19835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:35.691506 19955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ec53b7046604f9a9294453519132be4: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:35.692313 19955 log.cc:824] T 00000000000000000000000000000000 P 4ec53b7046604f9a9294453519132be4: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:35.694411 19955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ec53b7046604f9a9294453519132be4: No bootstrap required, opened a new log
I20260807 06:22:35.696429 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:35.683438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33300 (local address 127.15.254.62:46491)
0807 06:22:35.683951 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:35.683968 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:35.684012 (+ 44us) server_negotiation.cc:407] Connection header received
0807 06:22:35.684738 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:35.684756 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:35.685102 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:35.685457 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:35.686288 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:35.687287 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:35.688155 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:35.688700 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:35.688880 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:35.691439 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:35.691444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:35.691458 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:35.691488 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:35.693410 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:35.693891 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:35.693896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:35.693901 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:35.694029 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:35.694332 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:35.694349 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:35.694350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:35.694843 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:35.695062 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:35.695369 (+ 307us) server_negotiation.cc:299] Negotiation successful
0807 06:22:35.695683 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":158,"threads_started":1}
I20260807 06:22:35.698273 19955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ec53b7046604f9a9294453519132be4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec53b7046604f9a9294453519132be4" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 46491 } }
I20260807 06:22:35.698552 19955 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ec53b7046604f9a9294453519132be4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:35.698645 19955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ec53b7046604f9a9294453519132be4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec53b7046604f9a9294453519132be4, State: Initialized, Role: FOLLOWER
I20260807 06:22:35.699146 19955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ec53b7046604f9a9294453519132be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec53b7046604f9a9294453519132be4" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 46491 } }
I20260807 06:22:35.699318 19955 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ec53b7046604f9a9294453519132be4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:22:35.699411 19955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ec53b7046604f9a9294453519132be4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:22:35.699543 19955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ec53b7046604f9a9294453519132be4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:35.700621 19955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ec53b7046604f9a9294453519132be4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec53b7046604f9a9294453519132be4" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 46491 } }
I20260807 06:22:35.701007 19955 leader_election.cc:302] T 00000000000000000000000000000000 P 4ec53b7046604f9a9294453519132be4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ec53b7046604f9a9294453519132be4; no voters:
I20260807 06:22:35.701383 19955 leader_election.cc:288] T 00000000000000000000000000000000 P 4ec53b7046604f9a9294453519132be4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:22:35.701548 19962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ec53b7046604f9a9294453519132be4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:35.701857 19962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ec53b7046604f9a9294453519132be4 [term 1 LEADER]: Becoming Leader. State: Replica: 4ec53b7046604f9a9294453519132be4, State: Running, Role: LEADER
I20260807 06:22:35.702304 19962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ec53b7046604f9a9294453519132be4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec53b7046604f9a9294453519132be4" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 46491 } }
I20260807 06:22:35.702976 19955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ec53b7046604f9a9294453519132be4 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:22:35.704216 19963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ec53b7046604f9a9294453519132be4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ec53b7046604f9a9294453519132be4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec53b7046604f9a9294453519132be4" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 46491 } } }
I20260807 06:22:35.704358 19963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ec53b7046604f9a9294453519132be4 [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:35.704595 19965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ec53b7046604f9a9294453519132be4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ec53b7046604f9a9294453519132be4. Latest consensus state: current_term: 1 leader_uuid: "4ec53b7046604f9a9294453519132be4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec53b7046604f9a9294453519132be4" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 46491 } } }
I20260807 06:22:35.704701 19965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ec53b7046604f9a9294453519132be4 [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:35.705057 19974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:22:35.708381 19974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:22:35.714463 19974 catalog_manager.cc:1347] Generated new cluster ID: 3014bf1df75643308a9e67ad80abb771
I20260807 06:22:35.714541 19974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:22:35.737655 19974 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:22:35.738713 19974 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:22:35.751782 19974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ec53b7046604f9a9294453519132be4: Generated new TSK 0
I20260807 06:22:35.752875 19974 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
I20260807 06:22:35.797806 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:0
--local_ip_for_outbound_sockets=127.15.254.1
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:46491
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:38959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:35.937155 19988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:35.937433 19988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:35.937475 19988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:35.942059 19988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:35.942154 19988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:35.942219 19988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:22:35.948243 19988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:38959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:46491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:35.949535 19988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:35.950619 19988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:35.960716 19993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:35.960736 19994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:35.960721 19996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:35.961140 19988 server_base.cc:1034] running on GCE node
I20260807 06:22:35.961630 19988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:35.962347 19988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:35.963550 19988 hybrid_clock.cc:648] HybridClock initialized: now 1786083755963535 us; error 39 us; skew 500 ppm
Aug 07 06:22:35 dist-test-slave-44d2 krb5kdc[19835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083755, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:35.966619 19988 init.cc:377] Logged in from keytab as kudu/127.15.254.1@KRBTEST.COM (short username kudu)
I20260807 06:22:35.967509 19988 webserver.cc:460] Webserver started at http://127.15.254.1:39841/ using document root <none> and password file <none>
I20260807 06:22:35.968061 19988 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:35.968123 19988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:35.968295 19988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:35.970441 19988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/data/instance:
uuid: "23d659bb21e04e27982c9f9b89b499db"
format_stamp: "Formatted at 2026-08-07 06:22:35 on dist-test-slave-44d2"
server_key: "84fca6c80c1c8a7136b2343f61590479"
server_key_iv: "112e0283851afc9eb88ab5fa1c76ad84"
server_key_version: "encryptionkey@0"
I20260807 06:22:35.970989 19988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance:
uuid: "23d659bb21e04e27982c9f9b89b499db"
format_stamp: "Formatted at 2026-08-07 06:22:35 on dist-test-slave-44d2"
server_key: "84fca6c80c1c8a7136b2343f61590479"
server_key_iv: "112e0283851afc9eb88ab5fa1c76ad84"
server_key_version: "encryptionkey@0"
I20260807 06:22:35.974918 19988 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:22:35.977432 20004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:35.978749 19988 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 06:22:35.978885 19988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "23d659bb21e04e27982c9f9b89b499db"
format_stamp: "Formatted at 2026-08-07 06:22:35 on dist-test-slave-44d2"
server_key: "84fca6c80c1c8a7136b2343f61590479"
server_key_iv: "112e0283851afc9eb88ab5fa1c76ad84"
server_key_version: "encryptionkey@0"
I20260807 06:22:35.978998 19988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:36.005481 19988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:36.009300 19988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:36.009537 19988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:36.010226 19988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:36.011161 19988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:36.011211 19988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:36.011283 19988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:36.011314 19988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:36.021020 19988 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:42979
I20260807 06:22:36.021051 20172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:42979 every 8 connection(s)
I20260807 06:22:36.022068 19988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:22:36.026734 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 19988
I20260807 06:22:36.026862 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance
I20260807 06:22:36.027223 16376 external_mini_cluster.cc:1428] Setting key aed68ce22636a05b1c981e154b732e53
Aug 07 06:22:36 dist-test-slave-44d2 krb5kdc[19835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083755, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.254.2@KRBTEST.COM; defaulting to no policy
I20260807 06:22:36.038856 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:36.024106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:40929 (local address 127.15.254.62:46491)
0807 06:22:36.024257 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:36.024261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:36.025301 (+ 1040us) server_negotiation.cc:407] Connection header received
0807 06:22:36.026854 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:36.026860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:36.026933 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:36.027029 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:36.028630 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:36.029290 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:36.031121 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:36.031376 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:36.031521 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:36.034562 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:36.034566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:36.034568 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:36.034593 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:36.036440 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:36.037107 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:36.037111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:36.037113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:36.037210 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:36.037610 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:36.037613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:36.037615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:36.037821 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:36.037945 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:36.038502 (+ 557us) server_negotiation.cc:299] Negotiation successful
0807 06:22:36.038654 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 06:22:36.039942 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:36.024645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:46491 (local address 127.15.254.1:40929)
0807 06:22:36.025143 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:36.025190 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:36.026558 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:36.027265 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:36.027275 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:36.027795 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:36.028440 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:36.028462 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:36.029434 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:36.029438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:36.030952 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:36.030968 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:36.031571 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:36.031575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:36.031759 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:36.032525 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:36.032552 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:36.034357 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:36.036626 (+ 2269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:36.036634 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:36.036652 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:36.036978 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:36.037317 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:36.037321 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:36.037323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:36.037490 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:36.037943 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:36.037951 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:36.038229 (+ 278us) client_negotiation.cc:769] Sending connection context
0807 06:22:36.038441 (+ 212us) client_negotiation.cc:240] Negotiation successful
0807 06:22:36.038819 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":131,"threads_started":1}
I20260807 06:22:36.041447 20173 heartbeater.cc:347] Connected to a master server at 127.15.254.62:46491
I20260807 06:22:36.041814 20173 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.2@KRBTEST.COM" created.
I20260807 06:22:36.042384 20173 heartbeater.cc:511] Master 127.15.254.62:46491 requested a full tablet report, sending...
I20260807 06:22:36.044060 19908 ts_manager.cc:194] Registered new tserver with Master: 23d659bb21e04e27982c9f9b89b499db (127.15.254.1:42979)
I20260807 06:22:36.045385 19908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.1@KRBTEST.COM'} at 127.15.254.1:40929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
I20260807 06:22:36.060885 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:0
--local_ip_for_outbound_sockets=127.15.254.2
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:46491
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:38959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:36.186060 20180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:36.186329 20180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:36.186374 20180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:36.190443 20180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:36.190507 20180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:36.190567 20180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:22:36.196110 20180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:38959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:46491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:36.197319 20180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:36.198376 20180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:36.208607 20192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:36.208617 20188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:36.208871 20180 server_base.cc:1034] running on GCE node
W20260807 06:22:36.208617 20187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:36.209407 20180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:36.210114 20180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:36.211328 20180 hybrid_clock.cc:648] HybridClock initialized: now 1786083756211291 us; error 47 us; skew 500 ppm
Aug 07 06:22:36 dist-test-slave-44d2 krb5kdc[19835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083756, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:36.214351 20180 init.cc:377] Logged in from keytab as kudu/127.15.254.2@KRBTEST.COM (short username kudu)
I20260807 06:22:36.215277 20180 webserver.cc:460] Webserver started at http://127.15.254.2:35539/ using document root <none> and password file <none>
I20260807 06:22:36.215866 20180 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:36.215960 20180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:36.216200 20180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:36.218508 20180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/data/instance:
uuid: "e943eaf752b642fea1aeda75dc905262"
format_stamp: "Formatted at 2026-08-07 06:22:36 on dist-test-slave-44d2"
server_key: "5c431bd765e877d3aa595c1ec8f0eb31"
server_key_iv: "4e3d0f6c4ac6a5d1126efd0219784f07"
server_key_version: "encryptionkey@0"
I20260807 06:22:36.219074 20180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance:
uuid: "e943eaf752b642fea1aeda75dc905262"
format_stamp: "Formatted at 2026-08-07 06:22:36 on dist-test-slave-44d2"
server_key: "5c431bd765e877d3aa595c1ec8f0eb31"
server_key_iv: "4e3d0f6c4ac6a5d1126efd0219784f07"
server_key_version: "encryptionkey@0"
I20260807 06:22:36.223146 20180 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:22:36.225924 20204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:36.227202 20180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 06:22:36.227355 20180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "e943eaf752b642fea1aeda75dc905262"
format_stamp: "Formatted at 2026-08-07 06:22:36 on dist-test-slave-44d2"
server_key: "5c431bd765e877d3aa595c1ec8f0eb31"
server_key_iv: "4e3d0f6c4ac6a5d1126efd0219784f07"
server_key_version: "encryptionkey@0"
I20260807 06:22:36.227480 20180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:36.238061 20180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:36.241294 20180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:36.241518 20180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:36.242146 20180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:36.243045 20180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:36.243124 20180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:36.243199 20180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:36.243260 20180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:36.252746 20180 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:40641
I20260807 06:22:36.252769 20356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:40641 every 8 connection(s)
I20260807 06:22:36.253683 20180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:22:36.258908 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 20180
I20260807 06:22:36.259078 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance
I20260807 06:22:36.259366 16376 external_mini_cluster.cc:1428] Setting key 766931fd4fc25df980737634e2dac11b
Aug 07 06:22:36 dist-test-slave-44d2 krb5kdc[19835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083756, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:36.269790 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:36.255557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:50163 (local address 127.15.254.62:46491)
0807 06:22:36.255723 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:36.255728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:36.257135 (+ 1407us) server_negotiation.cc:407] Connection header received
0807 06:22:36.258164 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:36.258167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:36.258218 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:36.258298 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:36.260001 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:36.260698 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:36.262037 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:36.262462 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:36.262597 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:36.265436 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:36.265443 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:36.265456 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:36.265492 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:36.267393 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:36.267949 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:36.267952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:36.267957 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:36.268060 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:36.268418 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:36.268424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:36.268427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:36.268742 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:36.268858 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:36.269400 (+ 542us) server_negotiation.cc:299] Negotiation successful
0807 06:22:36.269566 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 06:22:36.270627 20360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:36.256013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:46491 (local address 127.15.254.2:50163)
0807 06:22:36.256972 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:36.257018 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:36.257939 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:36.258523 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:36.258530 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:36.259067 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:36.259713 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:36.259738 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:36.260862 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:36.260865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:36.261892 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:36.261904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:36.262647 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:36.262650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:36.262823 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:36.263578 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:36.263597 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:36.265246 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:36.267534 (+ 2288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:36.267539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:36.267553 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:36.267838 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:36.268169 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:36.268172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:36.268173 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:36.268316 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:36.268855 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:36.268860 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:36.269126 (+ 266us) client_negotiation.cc:769] Sending connection context
0807 06:22:36.269300 (+ 174us) client_negotiation.cc:240] Negotiation successful
0807 06:22:36.269593 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":619,"threads_started":1}
WARNING: no policy specified for kudu/127.15.254.3@KRBTEST.COM; defaulting to no policy
I20260807 06:22:36.271993 20357 heartbeater.cc:347] Connected to a master server at 127.15.254.62:46491
I20260807 06:22:36.272253 20357 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:36.272850 20357 heartbeater.cc:511] Master 127.15.254.62:46491 requested a full tablet report, sending...
I20260807 06:22:36.274077 19908 ts_manager.cc:194] Registered new tserver with Master: e943eaf752b642fea1aeda75dc905262 (127.15.254.2:40641)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.3@KRBTEST.COM" created.
I20260807 06:22:36.274891 19908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.2@KRBTEST.COM'} at 127.15.254.2:50163
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
I20260807 06:22:36.291417 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:0
--local_ip_for_outbound_sockets=127.15.254.3
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:46491
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:38959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:36.408844 20367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:36.409126 20367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:36.409214 20367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:36.413218 20367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:36.413296 20367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:36.413399 20367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:22:36.418845 20367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:38959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:46491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:36.419920 20367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:36.421096 20367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:36.431671 20377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:36.431766 20379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:36.431738 20367 server_base.cc:1034] running on GCE node
W20260807 06:22:36.431667 20376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:36.432453 20367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:36.433279 20367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:36.434442 20367 hybrid_clock.cc:648] HybridClock initialized: now 1786083756434428 us; error 26 us; skew 500 ppm
Aug 07 06:22:36 dist-test-slave-44d2 krb5kdc[19835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083756, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:36.437449 20367 init.cc:377] Logged in from keytab as kudu/127.15.254.3@KRBTEST.COM (short username kudu)
I20260807 06:22:36.438438 20367 webserver.cc:460] Webserver started at http://127.15.254.3:43695/ using document root <none> and password file <none>
I20260807 06:22:36.438989 20367 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:36.439051 20367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:36.439221 20367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:36.441463 20367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/data/instance:
uuid: "07ca0133d2b74b43b34262f2a31b23ec"
format_stamp: "Formatted at 2026-08-07 06:22:36 on dist-test-slave-44d2"
server_key: "646fc5ae9ea94425a4325f4c9185b7b5"
server_key_iv: "6e53bbfa63324ae95dad526846b08cf3"
server_key_version: "encryptionkey@0"
I20260807 06:22:36.442004 20367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance:
uuid: "07ca0133d2b74b43b34262f2a31b23ec"
format_stamp: "Formatted at 2026-08-07 06:22:36 on dist-test-slave-44d2"
server_key: "646fc5ae9ea94425a4325f4c9185b7b5"
server_key_iv: "6e53bbfa63324ae95dad526846b08cf3"
server_key_version: "encryptionkey@0"
I20260807 06:22:36.445729 20367 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:22:36.448311 20391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:36.450213 20367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:22:36.450322 20367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "07ca0133d2b74b43b34262f2a31b23ec"
format_stamp: "Formatted at 2026-08-07 06:22:36 on dist-test-slave-44d2"
server_key: "646fc5ae9ea94425a4325f4c9185b7b5"
server_key_iv: "6e53bbfa63324ae95dad526846b08cf3"
server_key_version: "encryptionkey@0"
I20260807 06:22:36.450413 20367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:36.492069 20367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:36.495599 20367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:36.495765 20367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:36.496348 20367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:36.497259 20367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:36.497308 20367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:36.497356 20367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:36.497378 20367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:36.507515 20367 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:38267
I20260807 06:22:36.507545 20553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:38267 every 8 connection(s)
I20260807 06:22:36.508534 20367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:22:36.509516 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 20367
I20260807 06:22:36.509657 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance
I20260807 06:22:36.510038 16376 external_mini_cluster.cc:1428] Setting key 4e45ef84b4836e0f8e187566bbaf9d9f
Aug 07 06:22:36 dist-test-slave-44d2 krb5kdc[19835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083756, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:36.522950 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:36.511088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:39491 (local address 127.15.254.62:46491)
0807 06:22:36.511242 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:36.511247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:36.512049 (+ 802us) server_negotiation.cc:407] Connection header received
0807 06:22:36.513652 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:36.513655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:36.513710 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:36.513788 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:36.515014 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:36.515549 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:36.516750 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:36.516977 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:36.517096 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:36.519560 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:36.519563 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:36.519566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:36.519590 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:36.521117 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:36.521611 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:36.521613 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:36.521615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:36.521693 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:36.521996 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:36.521998 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:36.521999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:36.522180 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:36.522261 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:36.522730 (+ 469us) server_negotiation.cc:299] Negotiation successful
0807 06:22:36.522828 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 06:22:36.524030 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:36.511395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:46491 (local address 127.15.254.3:39491)
0807 06:22:36.511909 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:36.511954 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:36.513427 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:36.513971 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:36.513978 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:36.514357 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:36.514853 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:36.514867 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:36.515679 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:36.515683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:36.516618 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:36.516628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:36.517148 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:36.517152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:36.517322 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:36.517900 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:36.517917 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:36.519396 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:36.521228 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:36.521233 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:36.521245 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:36.521503 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:36.521777 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:36.521780 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:36.521781 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:36.521914 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:36.522277 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:36.522282 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:36.522504 (+ 222us) client_negotiation.cc:769] Sending connection context
0807 06:22:36.522682 (+ 178us) client_negotiation.cc:240] Negotiation successful
0807 06:22:36.523088 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":131,"threads_started":1}
I20260807 06:22:36.525133 20554 heartbeater.cc:347] Connected to a master server at 127.15.254.62:46491
I20260807 06:22:36.525382 20554 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:36.525882 20554 heartbeater.cc:511] Master 127.15.254.62:46491 requested a full tablet report, sending...
I20260807 06:22:36.526889 19908 ts_manager.cc:194] Registered new tserver with Master: 07ca0133d2b74b43b34262f2a31b23ec (127.15.254.3:38267)
I20260807 06:22:36.527455 19908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.3@KRBTEST.COM'} at 127.15.254.3:39491
I20260807 06:22:36.536439 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 06:22:36.553143 19960 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:22:36.551535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33312 (local address 127.15.254.62:46491)
0807 06:22:36.551731 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:36.551743 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:36.551824 (+ 81us) server_negotiation.cc:407] Connection header received
0807 06:22:36.552009 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:36.552012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:36.552060 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:36.552132 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:36.552947 (+ 815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33312: BlockingRecv error: recv got EOF from 127.0.0.1:33312 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260807 06:22:36.554862 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 19988
I20260807 06:22:36.571055 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 20180
I20260807 06:22:36.584620 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 20367
I20260807 06:22:36.597636 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 19858
2026-08-07T06:22:36Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2468 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:22:36 dist-test-slave-44d2 krb5kdc[20570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:22:36 dist-test-slave-44d2 krb5kdc[20570](info): set up 2 sockets
Aug 07 06:22:36 dist-test-slave-44d2 krb5kdc[20570](info): commencing operation
krb5kdc: starting...
W20260807 06:22:37.834368 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.209s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:22:37 dist-test-slave-44d2 krb5kdc[20570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:22:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:22:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:22:37.967710 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:45119
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:35827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:45119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:38.085388 20589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:38.085736 20589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:38.085815 20589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:38.089881 20589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:22:38.089975 20589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:38.090018 20589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:38.090052 20589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:22:38.090076 20589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:22:38.095845 20589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:45119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:45119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:38.097028 20589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:38.098109 20589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:38.107458 20596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:38.107481 20599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:38.107569 20589 server_base.cc:1034] running on GCE node
W20260807 06:22:38.107476 20595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:38.108316 20589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:38.109526 20589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:38.110731 20589 hybrid_clock.cc:648] HybridClock initialized: now 1786083758110712 us; error 27 us; skew 500 ppm
Aug 07 06:22:38 dist-test-slave-44d2 krb5kdc[20570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083758, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:38.113752 20589 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:22:38.114756 20589 webserver.cc:460] Webserver started at http://127.15.254.62:46183/ using document root <none> and password file <none>
I20260807 06:22:38.115370 20589 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:38.115464 20589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:38.115715 20589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:38.117879 20589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "8b018bd3f99c4e4c8d7841998362699d"
format_stamp: "Formatted at 2026-08-07 06:22:38 on dist-test-slave-44d2"
server_key: "f7f5d2dfbeeba669b3e774a1d6dedf7e"
server_key_iv: "6c4e7279a694c96f48578c41111f050b"
server_key_version: "encryptionkey@0"
I20260807 06:22:38.118505 20589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "8b018bd3f99c4e4c8d7841998362699d"
format_stamp: "Formatted at 2026-08-07 06:22:38 on dist-test-slave-44d2"
server_key: "f7f5d2dfbeeba669b3e774a1d6dedf7e"
server_key_iv: "6c4e7279a694c96f48578c41111f050b"
server_key_version: "encryptionkey@0"
I20260807 06:22:38.122390 20589 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:22:38.124905 20609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:38.126277 20589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:22:38.126434 20589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "8b018bd3f99c4e4c8d7841998362699d"
format_stamp: "Formatted at 2026-08-07 06:22:38 on dist-test-slave-44d2"
server_key: "f7f5d2dfbeeba669b3e774a1d6dedf7e"
server_key_iv: "6c4e7279a694c96f48578c41111f050b"
server_key_version: "encryptionkey@0"
I20260807 06:22:38.126549 20589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:38.141289 20589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:38.145097 20589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:38.145332 20589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:38.152923 20589 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:45119
I20260807 06:22:38.153330 20683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:45119 every 8 connection(s)
I20260807 06:22:38.153968 20589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:22:38.155656 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 20589
I20260807 06:22:38.155781 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:22:38.156096 16376 external_mini_cluster.cc:1428] Setting key dddff8f594c18c4399cd5e8bfcf4f554
I20260807 06:22:38.157414 20684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:38.163237 20684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b018bd3f99c4e4c8d7841998362699d: Bootstrap starting.
Aug 07 06:22:38 dist-test-slave-44d2 krb5kdc[20570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:38.166007 20684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b018bd3f99c4e4c8d7841998362699d: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:38.166786 20684 log.cc:824] T 00000000000000000000000000000000 P 8b018bd3f99c4e4c8d7841998362699d: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:38.169248 20684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b018bd3f99c4e4c8d7841998362699d: No bootstrap required, opened a new log
I20260807 06:22:38.171202 20691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:38.157565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35530 (local address 127.15.254.62:45119)
0807 06:22:38.158008 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:38.158029 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:38.158064 (+ 35us) server_negotiation.cc:407] Connection header received
0807 06:22:38.158709 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:38.158726 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:38.159058 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:38.159409 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:38.160254 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:38.161343 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:38.162271 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:38.162764 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:38.162944 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:38.165875 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:38.165880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:38.165895 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:38.165925 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:38.167982 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:38.168532 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:38.168539 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:38.168552 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:38.168680 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:38.168992 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:38.168995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:38.168996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:38.169434 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:38.169605 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:38.169920 (+ 315us) server_negotiation.cc:299] Negotiation successful
0807 06:22:38.170263 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":141,"threads_started":1}
I20260807 06:22:38.172756 20684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b018bd3f99c4e4c8d7841998362699d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b018bd3f99c4e4c8d7841998362699d" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 45119 } }
I20260807 06:22:38.173053 20684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b018bd3f99c4e4c8d7841998362699d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:38.173146 20684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b018bd3f99c4e4c8d7841998362699d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b018bd3f99c4e4c8d7841998362699d, State: Initialized, Role: FOLLOWER
I20260807 06:22:38.173671 20684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b018bd3f99c4e4c8d7841998362699d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b018bd3f99c4e4c8d7841998362699d" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 45119 } }
I20260807 06:22:38.173835 20684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b018bd3f99c4e4c8d7841998362699d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:22:38.173924 20684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b018bd3f99c4e4c8d7841998362699d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:22:38.174039 20684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b018bd3f99c4e4c8d7841998362699d [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:38.174479 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:22:38.175114 20684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b018bd3f99c4e4c8d7841998362699d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b018bd3f99c4e4c8d7841998362699d" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 45119 } }
I20260807 06:22:38.175463 20684 leader_election.cc:302] T 00000000000000000000000000000000 P 8b018bd3f99c4e4c8d7841998362699d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b018bd3f99c4e4c8d7841998362699d; no voters:
I20260807 06:22:38.175794 20684 leader_election.cc:288] T 00000000000000000000000000000000 P 8b018bd3f99c4e4c8d7841998362699d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:22:38.175917 20693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b018bd3f99c4e4c8d7841998362699d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:38.176196 20693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b018bd3f99c4e4c8d7841998362699d [term 1 LEADER]: Becoming Leader. State: Replica: 8b018bd3f99c4e4c8d7841998362699d, State: Running, Role: LEADER
I20260807 06:22:38.176622 20693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b018bd3f99c4e4c8d7841998362699d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b018bd3f99c4e4c8d7841998362699d" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 45119 } }
I20260807 06:22:38.177223 20684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b018bd3f99c4e4c8d7841998362699d [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:22:38.178695 20695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b018bd3f99c4e4c8d7841998362699d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b018bd3f99c4e4c8d7841998362699d. Latest consensus state: current_term: 1 leader_uuid: "8b018bd3f99c4e4c8d7841998362699d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b018bd3f99c4e4c8d7841998362699d" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 45119 } } }
I20260807 06:22:38.178828 20695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b018bd3f99c4e4c8d7841998362699d [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:38.179204 20710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:22:38.179165 20694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b018bd3f99c4e4c8d7841998362699d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b018bd3f99c4e4c8d7841998362699d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b018bd3f99c4e4c8d7841998362699d" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 45119 } } }
I20260807 06:22:38.179339 20694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b018bd3f99c4e4c8d7841998362699d [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:38.182869 20710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:22:38.189214 20710 catalog_manager.cc:1347] Generated new cluster ID: 9704e605019645aab53f461a2bf1cf79
I20260807 06:22:38.189297 20710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:22:38.194188 20710 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:22:38.195240 20710 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:22:38.199954 20710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b018bd3f99c4e4c8d7841998362699d: Generated new TSK 0
I20260807 06:22:38.200744 20710 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:22:38.202963 20691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:38.197186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35538 (local address 127.15.254.62:45119)
0807 06:22:38.197292 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:38.197297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:38.197608 (+ 311us) server_negotiation.cc:407] Connection header received
0807 06:22:38.197743 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:38.197746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:38.197794 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:38.197896 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:38.198547 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:38.199285 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:38.200584 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:38.200935 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:38.201055 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:38.202019 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:38.202022 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:38.202024 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:22:38.202498 (+ 474us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:38.202529 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:22:38.202546 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:38.202629 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:38.202755 (+ 126us) server_negotiation.cc:299] Negotiation successful
0807 06:22:38.202815 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260807 06:22:38.207791 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 20589
2026-08-07T06:22:38Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1615 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:22:38 dist-test-slave-44d2 krb5kdc[20735](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:22:38 dist-test-slave-44d2 krb5kdc[20735](info): set up 2 sockets
Aug 07 06:22:38 dist-test-slave-44d2 krb5kdc[20735](info): commencing operation
krb5kdc: starting...
W20260807 06:22:39.439251 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:22:39 dist-test-slave-44d2 krb5kdc[20735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:22:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:22:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:22:39.577453 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:39331
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:34371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:39331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:39.708752 20754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:39.709085 20754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:39.709172 20754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:39.713280 20754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:22:39.713368 20754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:39.713407 20754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:39.713474 20754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:22:39.713513 20754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:22:39.719218 20754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:34371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:39331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:39331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:39.720363 20754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:39.721479 20754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:39.730636 20764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:39.730639 20761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:39.730695 20754 server_base.cc:1034] running on GCE node
W20260807 06:22:39.730639 20762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:39.731712 20754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:39.732868 20754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:39.734069 20754 hybrid_clock.cc:648] HybridClock initialized: now 1786083759734062 us; error 33 us; skew 500 ppm
Aug 07 06:22:39 dist-test-slave-44d2 krb5kdc[20735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083759, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:39.737111 20754 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:22:39.738126 20754 webserver.cc:460] Webserver started at http://127.15.254.62:38363/ using document root <none> and password file <none>
I20260807 06:22:39.738715 20754 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:39.738813 20754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:39.739064 20754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:39.741223 20754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "f5470d41977448c5ba56d40d996ad1b8"
format_stamp: "Formatted at 2026-08-07 06:22:39 on dist-test-slave-44d2"
server_key: "389e4dd5affd8a1289ac27bc18f328ff"
server_key_iv: "57cd50f086cea1bb5b237e43aeadc965"
server_key_version: "encryptionkey@0"
I20260807 06:22:39.741832 20754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "f5470d41977448c5ba56d40d996ad1b8"
format_stamp: "Formatted at 2026-08-07 06:22:39 on dist-test-slave-44d2"
server_key: "389e4dd5affd8a1289ac27bc18f328ff"
server_key_iv: "57cd50f086cea1bb5b237e43aeadc965"
server_key_version: "encryptionkey@0"
I20260807 06:22:39.745858 20754 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:22:39.748376 20776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:39.749756 20754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 06:22:39.749920 20754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "f5470d41977448c5ba56d40d996ad1b8"
format_stamp: "Formatted at 2026-08-07 06:22:39 on dist-test-slave-44d2"
server_key: "389e4dd5affd8a1289ac27bc18f328ff"
server_key_iv: "57cd50f086cea1bb5b237e43aeadc965"
server_key_version: "encryptionkey@0"
I20260807 06:22:39.750046 20754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:39.770207 20754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:39.773655 20754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:39.773872 20754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:39.781476 20754 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:39331
I20260807 06:22:39.781785 20854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:39331 every 8 connection(s)
I20260807 06:22:39.782552 20754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:22:39.784518 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 20754
I20260807 06:22:39.784677 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:22:39.785004 16376 external_mini_cluster.cc:1428] Setting key 12b467ff85d7a038a3860d9632d902d5
I20260807 06:22:39.787030 20855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:39.792991 20855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5470d41977448c5ba56d40d996ad1b8: Bootstrap starting.
Aug 07 06:22:39 dist-test-slave-44d2 krb5kdc[20735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:39.795644 20855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5470d41977448c5ba56d40d996ad1b8: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:39.796417 20855 log.cc:824] T 00000000000000000000000000000000 P f5470d41977448c5ba56d40d996ad1b8: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:39.798465 20855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5470d41977448c5ba56d40d996ad1b8: No bootstrap required, opened a new log
I20260807 06:22:39.799278 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:39.786390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38638 (local address 127.15.254.62:39331)
0807 06:22:39.786851 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:39.786868 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:39.786908 (+ 40us) server_negotiation.cc:407] Connection header received
0807 06:22:39.787649 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:39.787668 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:39.788033 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:39.788379 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:39.789253 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:39.790261 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:39.791151 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:39.791623 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:39.791808 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:39.794478 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:39.794484 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:39.794496 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:39.794524 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:39.796291 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:39.796833 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:39.796838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:39.796842 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:39.796958 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:39.797334 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:39.797338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:39.797339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:39.797724 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:39.797930 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:39.798214 (+ 284us) server_negotiation.cc:299] Negotiation successful
0807 06:22:39.798536 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":137,"threads_started":1}
I20260807 06:22:39.802266 20855 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5470d41977448c5ba56d40d996ad1b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5470d41977448c5ba56d40d996ad1b8" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39331 } }
I20260807 06:22:39.802510 20855 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5470d41977448c5ba56d40d996ad1b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:39.802608 20855 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5470d41977448c5ba56d40d996ad1b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5470d41977448c5ba56d40d996ad1b8, State: Initialized, Role: FOLLOWER
I20260807 06:22:39.803030 20855 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5470d41977448c5ba56d40d996ad1b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5470d41977448c5ba56d40d996ad1b8" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39331 } }
I20260807 06:22:39.803188 20855 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5470d41977448c5ba56d40d996ad1b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:22:39.803278 20855 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5470d41977448c5ba56d40d996ad1b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:22:39.803400 20855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5470d41977448c5ba56d40d996ad1b8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:39.804371 20855 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5470d41977448c5ba56d40d996ad1b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5470d41977448c5ba56d40d996ad1b8" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39331 } }
I20260807 06:22:39.804791 20855 leader_election.cc:302] T 00000000000000000000000000000000 P f5470d41977448c5ba56d40d996ad1b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5470d41977448c5ba56d40d996ad1b8; no voters:
I20260807 06:22:39.805178 20855 leader_election.cc:288] T 00000000000000000000000000000000 P f5470d41977448c5ba56d40d996ad1b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:22:39.805334 20863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5470d41977448c5ba56d40d996ad1b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:39.805590 20863 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5470d41977448c5ba56d40d996ad1b8 [term 1 LEADER]: Becoming Leader. State: Replica: f5470d41977448c5ba56d40d996ad1b8, State: Running, Role: LEADER
I20260807 06:22:39.805923 20863 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5470d41977448c5ba56d40d996ad1b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5470d41977448c5ba56d40d996ad1b8" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39331 } }
I20260807 06:22:39.806497 20855 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5470d41977448c5ba56d40d996ad1b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:22:39.807821 20864 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5470d41977448c5ba56d40d996ad1b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5470d41977448c5ba56d40d996ad1b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5470d41977448c5ba56d40d996ad1b8" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39331 } } }
I20260807 06:22:39.807984 20864 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5470d41977448c5ba56d40d996ad1b8 [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:39.808277 20866 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5470d41977448c5ba56d40d996ad1b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5470d41977448c5ba56d40d996ad1b8. Latest consensus state: current_term: 1 leader_uuid: "f5470d41977448c5ba56d40d996ad1b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5470d41977448c5ba56d40d996ad1b8" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39331 } } }
I20260807 06:22:39.808393 20866 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5470d41977448c5ba56d40d996ad1b8 [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:39.808465 20879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:22:39.811335 20879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:22:39.816918 20879 catalog_manager.cc:1347] Generated new cluster ID: 2d25fdc493fb4c8e895183d9cf7b0f7c
I20260807 06:22:39.816991 20879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:22:39.833703 20879 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:22:39.834851 20879 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:22:39.849795 20879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5470d41977448c5ba56d40d996ad1b8: Generated new TSK 0
I20260807 06:22:39.850565 20879 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
I20260807 06:22:39.889349 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:0
--local_ip_for_outbound_sockets=127.15.254.1
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:39331
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:34371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:40.012321 20897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:40.012737 20897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:40.012791 20897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:40.016851 20897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:40.016908 20897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:40.016966 20897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:22:40.022373 20897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:34371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:39331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:40.023507 20897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:40.024586 20897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:40.034883 20906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:40.034970 20897 server_base.cc:1034] running on GCE node
W20260807 06:22:40.034883 20912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:40.034983 20907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:40.035619 20897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:40.036288 20897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:40.037497 20897 hybrid_clock.cc:648] HybridClock initialized: now 1786083760037461 us; error 53 us; skew 500 ppm
Aug 07 06:22:40 dist-test-slave-44d2 krb5kdc[20735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083760, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:40.040588 20897 init.cc:377] Logged in from keytab as kudu/127.15.254.1@KRBTEST.COM (short username kudu)
I20260807 06:22:40.041514 20897 webserver.cc:460] Webserver started at http://127.15.254.1:43397/ using document root <none> and password file <none>
I20260807 06:22:40.042100 20897 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:40.042197 20897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:40.042448 20897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:40.044550 20897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/data/instance:
uuid: "f4cc8365a8c24051bf7ce8ea82d48112"
format_stamp: "Formatted at 2026-08-07 06:22:40 on dist-test-slave-44d2"
server_key: "08200870f24f4e569fc55de7754494b1"
server_key_iv: "65df9db2dce0d2e6400498fc500fbec8"
server_key_version: "encryptionkey@0"
I20260807 06:22:40.045197 20897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance:
uuid: "f4cc8365a8c24051bf7ce8ea82d48112"
format_stamp: "Formatted at 2026-08-07 06:22:40 on dist-test-slave-44d2"
server_key: "08200870f24f4e569fc55de7754494b1"
server_key_iv: "65df9db2dce0d2e6400498fc500fbec8"
server_key_version: "encryptionkey@0"
I20260807 06:22:40.048924 20897 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:22:40.051323 20922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:40.052660 20897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 06:22:40.052796 20897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "f4cc8365a8c24051bf7ce8ea82d48112"
format_stamp: "Formatted at 2026-08-07 06:22:40 on dist-test-slave-44d2"
server_key: "08200870f24f4e569fc55de7754494b1"
server_key_iv: "65df9db2dce0d2e6400498fc500fbec8"
server_key_version: "encryptionkey@0"
I20260807 06:22:40.052907 20897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:40.064237 20897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:40.067806 20897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:40.068025 20897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:40.068778 20897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:40.069636 20897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:40.069706 20897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:40.069780 20897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:40.069831 20897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:40.078949 20897 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:36563
I20260807 06:22:40.079023 21082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:36563 every 8 connection(s)
I20260807 06:22:40.079869 20897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:22:40.086917 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 20897
I20260807 06:22:40.087072 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance
I20260807 06:22:40.087404 16376 external_mini_cluster.cc:1428] Setting key 220a225ad865647cb5ef77cd5f6ebe9b
Aug 07 06:22:40 dist-test-slave-44d2 krb5kdc[20735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083760, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:40.095355 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:40.081781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:34617 (local address 127.15.254.62:39331)
0807 06:22:40.081991 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:40.081996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:40.082935 (+ 939us) server_negotiation.cc:407] Connection header received
0807 06:22:40.084423 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:40.084427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:40.084509 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:40.084586 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:40.085806 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:40.086376 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:40.088096 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:40.088352 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:40.088472 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:40.091055 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:40.091059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:40.091062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:40.091098 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:40.092861 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:40.093536 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:40.093539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:40.093540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:40.093626 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:40.094047 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:40.094050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:40.094051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:40.094235 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:40.094314 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:40.095101 (+ 787us) server_negotiation.cc:299] Negotiation successful
0807 06:22:40.095230 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 06:22:40.096664 21089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:40.082202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39331 (local address 127.15.254.1:34617)
0807 06:22:40.082711 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:40.082758 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:40.084167 (+ 1409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:40.084767 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:40.084774 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:40.085152 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:40.085650 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:40.085663 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:40.086545 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:40.086548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:40.087956 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:40.087968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:40.088536 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:40.088541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:40.088716 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:40.089385 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:40.089402 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:40.090881 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:40.092996 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:40.093006 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:40.093025 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:40.093414 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:40.093732 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:40.093738 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:40.093743 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:40.093946 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:40.094349 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:40.094358 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:40.094781 (+ 423us) client_negotiation.cc:769] Sending connection context
0807 06:22:40.095027 (+ 246us) client_negotiation.cc:240] Negotiation successful
0807 06:22:40.095437 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":124,"threads_started":1}
I20260807 06:22:40.098053 21084 heartbeater.cc:347] Connected to a master server at 127.15.254.62:39331
I20260807 06:22:40.098309 21084 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:40.098834 21084 heartbeater.cc:511] Master 127.15.254.62:39331 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.254.2@KRBTEST.COM; defaulting to no policy
I20260807 06:22:40.100728 20801 ts_manager.cc:194] Registered new tserver with Master: f4cc8365a8c24051bf7ce8ea82d48112 (127.15.254.1:36563)
I20260807 06:22:40.102021 20801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.1@KRBTEST.COM'} at 127.15.254.1:34617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
I20260807 06:22:40.119612 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:0
--local_ip_for_outbound_sockets=127.15.254.2
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:39331
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:34371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:40.236516 21093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:40.236865 21093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:40.236954 21093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:40.241092 21093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:40.241196 21093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:40.241303 21093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:22:40.246773 21093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:34371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:39331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:40.247960 21093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:40.249094 21093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:40.260360 21101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:40.260339 21100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:40.260653 21093 server_base.cc:1034] running on GCE node
W20260807 06:22:40.260346 21103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:40.261229 21093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:40.262044 21093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:40.263255 21093 hybrid_clock.cc:648] HybridClock initialized: now 1786083760263230 us; error 35 us; skew 500 ppm
Aug 07 06:22:40 dist-test-slave-44d2 krb5kdc[20735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083760, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:40.266750 21093 init.cc:377] Logged in from keytab as kudu/127.15.254.2@KRBTEST.COM (short username kudu)
I20260807 06:22:40.267876 21093 webserver.cc:460] Webserver started at http://127.15.254.2:45519/ using document root <none> and password file <none>
I20260807 06:22:40.268617 21093 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:40.268731 21093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:40.268981 21093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:40.271175 21093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/data/instance:
uuid: "189e74eb54ed4298a2f8d7be8f22057c"
format_stamp: "Formatted at 2026-08-07 06:22:40 on dist-test-slave-44d2"
server_key: "0a42d9c71ea7b5af685c9df5ef322c87"
server_key_iv: "0d36b5bc999cf66c505969e0ba2013b7"
server_key_version: "encryptionkey@0"
I20260807 06:22:40.271773 21093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance:
uuid: "189e74eb54ed4298a2f8d7be8f22057c"
format_stamp: "Formatted at 2026-08-07 06:22:40 on dist-test-slave-44d2"
server_key: "0a42d9c71ea7b5af685c9df5ef322c87"
server_key_iv: "0d36b5bc999cf66c505969e0ba2013b7"
server_key_version: "encryptionkey@0"
I20260807 06:22:40.275908 21093 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 06:22:40.278700 21114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:40.279983 21093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260807 06:22:40.280134 21093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "189e74eb54ed4298a2f8d7be8f22057c"
format_stamp: "Formatted at 2026-08-07 06:22:40 on dist-test-slave-44d2"
server_key: "0a42d9c71ea7b5af685c9df5ef322c87"
server_key_iv: "0d36b5bc999cf66c505969e0ba2013b7"
server_key_version: "encryptionkey@0"
I20260807 06:22:40.280256 21093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:40.310302 21093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:40.313939 21093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:40.314178 21093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:40.314877 21093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:40.315785 21093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:40.315861 21093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:40.315936 21093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:40.315992 21093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:40.325381 21093 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:32791
I20260807 06:22:40.325412 21273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:32791 every 8 connection(s)
I20260807 06:22:40.326341 21093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:22:40.327230 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 21093
I20260807 06:22:40.327328 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance
I20260807 06:22:40.327615 16376 external_mini_cluster.cc:1428] Setting key 2068f3ed348d9f854276b7dfc51806ad
Aug 07 06:22:40 dist-test-slave-44d2 krb5kdc[20735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083760, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.254.3@KRBTEST.COM; defaulting to no policy
I20260807 06:22:40.342799 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:40.328648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:53583 (local address 127.15.254.62:39331)
0807 06:22:40.328798 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:40.328802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:40.329827 (+ 1025us) server_negotiation.cc:407] Connection header received
0807 06:22:40.332025 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:40.332029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:40.332089 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:40.332167 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:40.333477 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:40.334336 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:40.335526 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:40.335931 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:40.336077 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:40.338807 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:40.338814 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:40.338819 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:40.338857 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:40.340547 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:40.341207 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:40.341210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:40.341211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:40.341291 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:40.341640 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:40.341642 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:40.341643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:40.341821 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:40.341900 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:40.342549 (+ 649us) server_negotiation.cc:299] Negotiation successful
0807 06:22:40.342667 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.3@KRBTEST.COM" created.
I20260807 06:22:40.343784 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:40.329008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39331 (local address 127.15.254.2:53583)
0807 06:22:40.329639 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:40.329700 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:40.331707 (+ 2007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:40.332347 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:40.332355 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:40.332826 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:40.333322 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:40.333336 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:40.334482 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:40.334485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:40.335395 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:40.335406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:40.336139 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:40.336143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:40.336303 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:40.337089 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:40.337107 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:40.338620 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:40.340705 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:40.340711 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:40.340725 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:40.341069 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:40.341399 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:40.341401 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:40.341403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:40.341549 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:40.341940 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:40.341946 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:40.342212 (+ 266us) client_negotiation.cc:769] Sending connection context
0807 06:22:40.342468 (+ 256us) client_negotiation.cc:240] Negotiation successful
0807 06:22:40.342781 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":177,"threads_started":1}
I20260807 06:22:40.345273 21274 heartbeater.cc:347] Connected to a master server at 127.15.254.62:39331
I20260807 06:22:40.345597 21274 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:40.346163 21274 heartbeater.cc:511] Master 127.15.254.62:39331 requested a full tablet report, sending...
I20260807 06:22:40.347349 20801 ts_manager.cc:194] Registered new tserver with Master: 189e74eb54ed4298a2f8d7be8f22057c (127.15.254.2:32791)
I20260807 06:22:40.348281 20801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.2@KRBTEST.COM'} at 127.15.254.2:53583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
I20260807 06:22:40.363628 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:0
--local_ip_for_outbound_sockets=127.15.254.3
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:39331
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:34371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:40.481024 21282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:40.481289 21282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:40.481333 21282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:40.485426 21282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:40.485491 21282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:40.485548 21282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:22:40.490875 21282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:34371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:39331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:40.491905 21282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:40.492992 21282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:40.502804 21291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:40.503059 21282 server_base.cc:1034] running on GCE node
W20260807 06:22:40.502827 21294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:40.502804 21289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:40.503647 21282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:40.504374 21282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:40.505543 21282 hybrid_clock.cc:648] HybridClock initialized: now 1786083760505527 us; error 28 us; skew 500 ppm
Aug 07 06:22:40 dist-test-slave-44d2 krb5kdc[20735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083760, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:40.508805 21282 init.cc:377] Logged in from keytab as kudu/127.15.254.3@KRBTEST.COM (short username kudu)
I20260807 06:22:40.509680 21282 webserver.cc:460] Webserver started at http://127.15.254.3:37095/ using document root <none> and password file <none>
I20260807 06:22:40.510241 21282 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:40.510300 21282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:40.510535 21282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:40.512717 21282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/data/instance:
uuid: "1be5d89709ea4a95871921cfa67638c4"
format_stamp: "Formatted at 2026-08-07 06:22:40 on dist-test-slave-44d2"
server_key: "6016f7bfcc0d808a6539a012156e54b0"
server_key_iv: "6d756b50dc605b95635b6cf5a1f3fec2"
server_key_version: "encryptionkey@0"
I20260807 06:22:40.513300 21282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance:
uuid: "1be5d89709ea4a95871921cfa67638c4"
format_stamp: "Formatted at 2026-08-07 06:22:40 on dist-test-slave-44d2"
server_key: "6016f7bfcc0d808a6539a012156e54b0"
server_key_iv: "6d756b50dc605b95635b6cf5a1f3fec2"
server_key_version: "encryptionkey@0"
I20260807 06:22:40.517163 21282 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 06:22:40.519594 21309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:40.520897 21282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 06:22:40.521029 21282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "1be5d89709ea4a95871921cfa67638c4"
format_stamp: "Formatted at 2026-08-07 06:22:40 on dist-test-slave-44d2"
server_key: "6016f7bfcc0d808a6539a012156e54b0"
server_key_iv: "6d756b50dc605b95635b6cf5a1f3fec2"
server_key_version: "encryptionkey@0"
I20260807 06:22:40.521133 21282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:40.542642 21282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:40.546425 21282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:40.546669 21282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:40.547351 21282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:40.548254 21282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:40.548305 21282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:40.548377 21282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:40.548419 21282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:40.558645 21282 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:34181
I20260807 06:22:40.558665 21460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:34181 every 8 connection(s)
I20260807 06:22:40.559600 21282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:22:40.561398 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 21282
I20260807 06:22:40.561527 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance
I20260807 06:22:40.561899 16376 external_mini_cluster.cc:1428] Setting key 4a3cdd95e627aaa04f138a383f447e9a
Aug 07 06:22:40 dist-test-slave-44d2 krb5kdc[20735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083760, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:40.573563 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:40.561425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:34927 (local address 127.15.254.62:39331)
0807 06:22:40.561618 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:40.561623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:40.562599 (+ 976us) server_negotiation.cc:407] Connection header received
0807 06:22:40.564122 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:40.564125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:40.564185 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:40.564264 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:40.565458 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:40.565988 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:40.567215 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:40.567423 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:40.567501 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:40.570253 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:40.570256 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:40.570258 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:40.570282 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:40.571660 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:40.572155 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:40.572158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:40.572159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:40.572240 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:40.572610 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:40.572613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:40.572614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:40.572788 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:40.572878 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:40.573332 (+ 454us) server_negotiation.cc:299] Negotiation successful
0807 06:22:40.573444 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260807 06:22:40.574357 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:40.561874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39331 (local address 127.15.254.3:34927)
0807 06:22:40.562424 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:40.562487 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:40.563925 (+ 1438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:40.564429 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:40.564436 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:40.564834 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:40.565329 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:40.565344 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:40.566138 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:40.566142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:40.567101 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:40.567112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:40.567713 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:40.567716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:40.567876 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:40.568557 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:40.568574 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:40.570088 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:40.571769 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:40.571774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:40.571785 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:40.572034 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:40.572331 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:40.572333 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:40.572334 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:40.572470 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:40.572883 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:40.572889 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:40.573116 (+ 227us) client_negotiation.cc:769] Sending connection context
0807 06:22:40.573267 (+ 151us) client_negotiation.cc:240] Negotiation successful
0807 06:22:40.573526 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":144,"threads_started":1}
I20260807 06:22:40.575448 21465 heartbeater.cc:347] Connected to a master server at 127.15.254.62:39331
I20260807 06:22:40.575708 21465 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:40.576198 21465 heartbeater.cc:511] Master 127.15.254.62:39331 requested a full tablet report, sending...
I20260807 06:22:40.577391 20801 ts_manager.cc:194] Registered new tserver with Master: 1be5d89709ea4a95871921cfa67638c4 (127.15.254.3:34181)
I20260807 06:22:40.577907 20801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.3@KRBTEST.COM'} at 127.15.254.3:34927
I20260807 06:22:40.588152 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 06:22:40 dist-test-slave-44d2 krb5kdc[20735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083760, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 07 06:22:40 dist-test-slave-44d2 krb5kdc[20735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083760, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:40.617300 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:40.608468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47956 (local address 127.15.254.62:39331)
0807 06:22:40.608652 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:40.608657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:40.608868 (+ 211us) server_negotiation.cc:407] Connection header received
0807 06:22:40.609059 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:40.609070 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:40.609121 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:40.609198 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:40.609906 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:40.610465 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:40.611322 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:40.611528 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:40.611610 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:40.614285 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:40.614289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:40.614291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:40.614316 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:40.615798 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:40.616181 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:40.616184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:40.616185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:40.616260 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:40.616555 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:40.616557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:40.616558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:40.616779 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:40.616887 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:40.617077 (+ 190us) server_negotiation.cc:299] Negotiation successful
0807 06:22:40.617174 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260807 06:22:40.617766 20801 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47956
I20260807 06:22:40.618733 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 20897
I20260807 06:22:40.630872 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 21093
I20260807 06:22:40.640647 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 21282
I20260807 06:22:40.653201 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 20754
2026-08-07T06:22:40Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2441 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:22:40 dist-test-slave-44d2 krb5kdc[21480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:22:40 dist-test-slave-44d2 krb5kdc[21480](info): set up 2 sockets
Aug 07 06:22:40 dist-test-slave-44d2 krb5kdc[21480](info): commencing operation
krb5kdc: starting...
W20260807 06:22:41.888114 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:22:41 dist-test-slave-44d2 krb5kdc[21480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:22:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:22:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:22:42.027712 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:37261
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:46063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:37261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:42.142570 21503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:42.142946 21503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:42.143029 21503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:42.147212 21503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:22:42.147334 21503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:42.147370 21503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:42.147409 21503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:22:42.147442 21503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:22:42.152968 21503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:46063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:37261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:37261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:42.154125 21503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:42.155229 21503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:42.164374 21508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:42.164427 21511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:42.164427 21514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:42.164506 21503 server_base.cc:1034] running on GCE node
I20260807 06:22:42.165267 21503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:42.166452 21503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:42.167650 21503 hybrid_clock.cc:648] HybridClock initialized: now 1786083762167636 us; error 27 us; skew 500 ppm
Aug 07 06:22:42 dist-test-slave-44d2 krb5kdc[21480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083762, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:42.170751 21503 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:22:42.171747 21503 webserver.cc:460] Webserver started at http://127.15.254.62:34615/ using document root <none> and password file <none>
I20260807 06:22:42.172334 21503 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:42.172397 21503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:42.172689 21503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:42.174763 21503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "d1ef1f5aeb7b4e40904c46ce4e8cce03"
format_stamp: "Formatted at 2026-08-07 06:22:42 on dist-test-slave-44d2"
server_key: "447bb4bf2d295ed1d4923472355f91f4"
server_key_iv: "0689671fd32405ee0961b319f5e36eee"
server_key_version: "encryptionkey@0"
I20260807 06:22:42.175354 21503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "d1ef1f5aeb7b4e40904c46ce4e8cce03"
format_stamp: "Formatted at 2026-08-07 06:22:42 on dist-test-slave-44d2"
server_key: "447bb4bf2d295ed1d4923472355f91f4"
server_key_iv: "0689671fd32405ee0961b319f5e36eee"
server_key_version: "encryptionkey@0"
I20260807 06:22:42.179235 21503 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 06:22:42.181758 21521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:42.183179 21503 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 06:22:42.183329 21503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "d1ef1f5aeb7b4e40904c46ce4e8cce03"
format_stamp: "Formatted at 2026-08-07 06:22:42 on dist-test-slave-44d2"
server_key: "447bb4bf2d295ed1d4923472355f91f4"
server_key_iv: "0689671fd32405ee0961b319f5e36eee"
server_key_version: "encryptionkey@0"
I20260807 06:22:42.183447 21503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:42.198436 21503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:42.201877 21503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:42.202134 21503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:42.210439 21503 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:37261
I20260807 06:22:42.210456 21605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:37261 every 8 connection(s)
I20260807 06:22:42.211449 21503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:22:42.214504 21606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:42.215626 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 21503
I20260807 06:22:42.215788 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:22:42.216104 16376 external_mini_cluster.cc:1428] Setting key 6e519e95070374fbfeb81e581f75bbde
I20260807 06:22:42.221971 21606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1ef1f5aeb7b4e40904c46ce4e8cce03: Bootstrap starting.
I20260807 06:22:42.224985 21606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1ef1f5aeb7b4e40904c46ce4e8cce03: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:42.225950 21606 log.cc:824] T 00000000000000000000000000000000 P d1ef1f5aeb7b4e40904c46ce4e8cce03: Log is configured to *not* fsync() on all Append() calls
Aug 07 06:22:42 dist-test-slave-44d2 krb5kdc[21480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:42.229036 21606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1ef1f5aeb7b4e40904c46ce4e8cce03: No bootstrap required, opened a new log
I20260807 06:22:42.233145 21610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:42.217626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52108 (local address 127.15.254.62:37261)
0807 06:22:42.218086 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:42.218109 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:42.218160 (+ 51us) server_negotiation.cc:407] Connection header received
0807 06:22:42.219093 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:42.219119 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:42.219585 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:42.219987 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:42.220750 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:42.222319 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:42.223178 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:42.223946 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:42.224118 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:42.227256 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:42.227264 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:42.227284 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:42.227319 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:42.229626 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:42.230091 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:42.230098 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:42.230104 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:42.230273 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:42.230549 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:42.230552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:42.230554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:42.231128 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:42.231344 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:42.231587 (+ 243us) server_negotiation.cc:299] Negotiation successful
0807 06:22:42.232008 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":120,"threads_started":1}
I20260807 06:22:42.234440 21606 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1ef1f5aeb7b4e40904c46ce4e8cce03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1ef1f5aeb7b4e40904c46ce4e8cce03" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37261 } }
I20260807 06:22:42.234727 21606 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1ef1f5aeb7b4e40904c46ce4e8cce03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:42.234810 21606 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1ef1f5aeb7b4e40904c46ce4e8cce03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1ef1f5aeb7b4e40904c46ce4e8cce03, State: Initialized, Role: FOLLOWER
I20260807 06:22:42.235268 21606 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1ef1f5aeb7b4e40904c46ce4e8cce03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1ef1f5aeb7b4e40904c46ce4e8cce03" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37261 } }
I20260807 06:22:42.235396 21606 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1ef1f5aeb7b4e40904c46ce4e8cce03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:22:42.235491 21606 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1ef1f5aeb7b4e40904c46ce4e8cce03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:22:42.235607 21606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1ef1f5aeb7b4e40904c46ce4e8cce03 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:42.236712 21606 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1ef1f5aeb7b4e40904c46ce4e8cce03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1ef1f5aeb7b4e40904c46ce4e8cce03" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37261 } }
I20260807 06:22:42.237083 21606 leader_election.cc:302] T 00000000000000000000000000000000 P d1ef1f5aeb7b4e40904c46ce4e8cce03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1ef1f5aeb7b4e40904c46ce4e8cce03; no voters:
I20260807 06:22:42.237418 21606 leader_election.cc:288] T 00000000000000000000000000000000 P d1ef1f5aeb7b4e40904c46ce4e8cce03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:22:42.237519 21612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1ef1f5aeb7b4e40904c46ce4e8cce03 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:42.237807 21612 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1ef1f5aeb7b4e40904c46ce4e8cce03 [term 1 LEADER]: Becoming Leader. State: Replica: d1ef1f5aeb7b4e40904c46ce4e8cce03, State: Running, Role: LEADER
I20260807 06:22:42.238147 21612 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1ef1f5aeb7b4e40904c46ce4e8cce03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1ef1f5aeb7b4e40904c46ce4e8cce03" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37261 } }
I20260807 06:22:42.238940 21606 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1ef1f5aeb7b4e40904c46ce4e8cce03 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:22:42.239977 21616 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ef1f5aeb7b4e40904c46ce4e8cce03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1ef1f5aeb7b4e40904c46ce4e8cce03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1ef1f5aeb7b4e40904c46ce4e8cce03" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37261 } } }
I20260807 06:22:42.240135 21616 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ef1f5aeb7b4e40904c46ce4e8cce03 [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:42.239995 21615 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ef1f5aeb7b4e40904c46ce4e8cce03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1ef1f5aeb7b4e40904c46ce4e8cce03. Latest consensus state: current_term: 1 leader_uuid: "d1ef1f5aeb7b4e40904c46ce4e8cce03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1ef1f5aeb7b4e40904c46ce4e8cce03" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37261 } } }
I20260807 06:22:42.240676 21615 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ef1f5aeb7b4e40904c46ce4e8cce03 [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:42.240978 21625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:22:42.244313 21625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:22:42.249760 21625 catalog_manager.cc:1347] Generated new cluster ID: ba72cf8dced149b9bed899ef05501907
I20260807 06:22:42.249830 21625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:22:42.266700 21625 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:22:42.268986 21625 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:22:42.284024 21625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1ef1f5aeb7b4e40904c46ce4e8cce03: Generated new TSK 0
I20260807 06:22:42.284899 21625 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
I20260807 06:22:42.327850 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:0
--local_ip_for_outbound_sockets=127.15.254.1
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:37261
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:46063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:42.451699 21638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:42.451951 21638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:42.452036 21638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:42.456424 21638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:42.456508 21638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:42.456624 21638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:22:42.461779 21638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:46063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:37261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:42.462750 21638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:42.463730 21638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:42.474274 21650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:42.474270 21646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:42.474270 21645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:42.475198 21638 server_base.cc:1034] running on GCE node
I20260807 06:22:42.475641 21638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:42.476359 21638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:42.477561 21638 hybrid_clock.cc:648] HybridClock initialized: now 1786083762477520 us; error 52 us; skew 500 ppm
Aug 07 06:22:42 dist-test-slave-44d2 krb5kdc[21480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083762, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:42.480569 21638 init.cc:377] Logged in from keytab as kudu/127.15.254.1@KRBTEST.COM (short username kudu)
I20260807 06:22:42.481593 21638 webserver.cc:460] Webserver started at http://127.15.254.1:36813/ using document root <none> and password file <none>
I20260807 06:22:42.482208 21638 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:42.482309 21638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:42.482575 21638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:42.484848 21638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-0/data/instance:
uuid: "cbed8307443642ce9ac427ba85d3c452"
format_stamp: "Formatted at 2026-08-07 06:22:42 on dist-test-slave-44d2"
server_key: "5531e8c6428a7aedd3eabb5ac334cfc1"
server_key_iv: "e0860d3ed18155b84a77bcfbceb1f00c"
server_key_version: "encryptionkey@0"
I20260807 06:22:42.485471 21638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance:
uuid: "cbed8307443642ce9ac427ba85d3c452"
format_stamp: "Formatted at 2026-08-07 06:22:42 on dist-test-slave-44d2"
server_key: "5531e8c6428a7aedd3eabb5ac334cfc1"
server_key_iv: "e0860d3ed18155b84a77bcfbceb1f00c"
server_key_version: "encryptionkey@0"
I20260807 06:22:42.489562 21638 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 06:22:42.492110 21659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:42.493491 21638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:22:42.493641 21638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "cbed8307443642ce9ac427ba85d3c452"
format_stamp: "Formatted at 2026-08-07 06:22:42 on dist-test-slave-44d2"
server_key: "5531e8c6428a7aedd3eabb5ac334cfc1"
server_key_iv: "e0860d3ed18155b84a77bcfbceb1f00c"
server_key_version: "encryptionkey@0"
I20260807 06:22:42.493757 21638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:42.519932 21638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:42.523186 21638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:42.523412 21638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:42.524096 21638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:42.524996 21638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:42.525082 21638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:42.525159 21638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:42.525203 21638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:42.534703 21638 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:38745
I20260807 06:22:42.534726 21823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:38745 every 8 connection(s)
I20260807 06:22:42.535594 21638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:22:42.536605 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 21638
I20260807 06:22:42.536734 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance
I20260807 06:22:42.537071 16376 external_mini_cluster.cc:1428] Setting key 7f1bc2ec68a050c7f9c09170e91ee5eb
Aug 07 06:22:42 dist-test-slave-44d2 krb5kdc[21480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083762, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.254.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.2@KRBTEST.COM" created.
I20260807 06:22:42.552430 21610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:42.537784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:52433 (local address 127.15.254.62:37261)
0807 06:22:42.538008 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:42.538012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:42.538896 (+ 884us) server_negotiation.cc:407] Connection header received
0807 06:22:42.540619 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:42.540623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:42.540679 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:42.540759 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:42.542459 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:42.543015 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:42.544926 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:42.545193 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:42.545290 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:42.548401 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:42.548406 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:42.548414 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:42.548441 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:42.550044 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:42.550716 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:42.550723 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:42.550725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:42.550817 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:42.551182 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:42.551186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:42.551187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:42.551387 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:42.551504 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:42.552107 (+ 603us) server_negotiation.cc:299] Negotiation successful
0807 06:22:42.552252 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260807 06:22:42.553810 21826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:42.538238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:37261 (local address 127.15.254.1:52433)
0807 06:22:42.538747 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:42.538803 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:42.540348 (+ 1545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:42.540938 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:42.540945 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:42.541521 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:42.542261 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:42.542286 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:42.543154 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:42.543161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:42.544788 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:42.544805 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:42.545385 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:42.545393 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:42.545663 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:42.546370 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:42.546400 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:42.548212 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:42.550192 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:42.550198 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:42.550216 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:42.550603 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:42.550915 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:42.550919 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:42.550921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:42.551080 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:42.551504 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:42.551512 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:42.551839 (+ 327us) client_negotiation.cc:769] Sending connection context
0807 06:22:42.552050 (+ 211us) client_negotiation.cc:240] Negotiation successful
0807 06:22:42.552417 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":155,"threads_started":1}
I20260807 06:22:42.555425 21824 heartbeater.cc:347] Connected to a master server at 127.15.254.62:37261
I20260807 06:22:42.555730 21824 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:42.556351 21824 heartbeater.cc:511] Master 127.15.254.62:37261 requested a full tablet report, sending...
I20260807 06:22:42.558137 21550 ts_manager.cc:194] Registered new tserver with Master: cbed8307443642ce9ac427ba85d3c452 (127.15.254.1:38745)
I20260807 06:22:42.559348 21550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.1@KRBTEST.COM'} at 127.15.254.1:52433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
I20260807 06:22:42.571282 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:0
--local_ip_for_outbound_sockets=127.15.254.2
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:37261
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:46063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:42.692847 21830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:42.693140 21830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:42.693184 21830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:42.697243 21830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:42.697302 21830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:42.697360 21830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:22:42.702682 21830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:46063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:37261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:42.703765 21830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:42.705031 21830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:42.715859 21837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:42.715936 21830 server_base.cc:1034] running on GCE node
W20260807 06:22:42.715885 21838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:42.715879 21841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:42.716549 21830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:42.717248 21830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:42.718477 21830 hybrid_clock.cc:648] HybridClock initialized: now 1786083762718439 us; error 51 us; skew 500 ppm
Aug 07 06:22:42 dist-test-slave-44d2 krb5kdc[21480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083762, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:42.721554 21830 init.cc:377] Logged in from keytab as kudu/127.15.254.2@KRBTEST.COM (short username kudu)
I20260807 06:22:42.722540 21830 webserver.cc:460] Webserver started at http://127.15.254.2:33029/ using document root <none> and password file <none>
I20260807 06:22:42.723147 21830 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:42.723246 21830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:42.723497 21830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:42.725711 21830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-1/data/instance:
uuid: "ee7c1849cd114aa8a5fdee67335586cf"
format_stamp: "Formatted at 2026-08-07 06:22:42 on dist-test-slave-44d2"
server_key: "d2e2ed5cfc87080020af5bd00e4104eb"
server_key_iv: "c56cdf8331507194321f7be6c70fd7cc"
server_key_version: "encryptionkey@0"
I20260807 06:22:42.726347 21830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance:
uuid: "ee7c1849cd114aa8a5fdee67335586cf"
format_stamp: "Formatted at 2026-08-07 06:22:42 on dist-test-slave-44d2"
server_key: "d2e2ed5cfc87080020af5bd00e4104eb"
server_key_iv: "c56cdf8331507194321f7be6c70fd7cc"
server_key_version: "encryptionkey@0"
I20260807 06:22:42.730242 21830 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:22:42.732944 21851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:42.734236 21830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 06:22:42.734382 21830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "ee7c1849cd114aa8a5fdee67335586cf"
format_stamp: "Formatted at 2026-08-07 06:22:42 on dist-test-slave-44d2"
server_key: "d2e2ed5cfc87080020af5bd00e4104eb"
server_key_iv: "c56cdf8331507194321f7be6c70fd7cc"
server_key_version: "encryptionkey@0"
I20260807 06:22:42.734509 21830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:42.750430 21830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:42.754065 21830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:42.754292 21830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:42.754936 21830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:42.755851 21830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:42.755926 21830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:42.755999 21830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:42.756049 21830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:42.766125 21830 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:46851
I20260807 06:22:42.766151 22002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:46851 every 8 connection(s)
I20260807 06:22:42.767093 21830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:22:42.768857 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 21830
I20260807 06:22:42.769003 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance
I20260807 06:22:42.769411 16376 external_mini_cluster.cc:1428] Setting key f8c8c776d6ad222a0a8571fa246b2ec1
Aug 07 06:22:42 dist-test-slave-44d2 krb5kdc[21480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083762, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.254.3@KRBTEST.COM; defaulting to no policy
I20260807 06:22:42.783233 21610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:42.769108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:43723 (local address 127.15.254.62:37261)
0807 06:22:42.769302 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:42.769308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:42.770391 (+ 1083us) server_negotiation.cc:407] Connection header received
0807 06:22:42.772031 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:42.772037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:42.772104 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:42.772204 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:42.773772 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:42.774310 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:42.776134 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:42.776349 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:42.776426 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:42.779515 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:42.779519 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:42.779521 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:42.779546 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:42.781072 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:42.781591 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:42.781596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:42.781598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:42.781687 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:42.782101 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:42.782105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:42.782107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:42.782299 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:42.782414 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:42.782922 (+ 508us) server_negotiation.cc:299] Negotiation successful
0807 06:22:42.783057 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.3@KRBTEST.COM" created.
I20260807 06:22:42.784339 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:42.769620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:37261 (local address 127.15.254.2:43723)
0807 06:22:42.770184 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:42.770266 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:42.771795 (+ 1529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:42.772412 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:42.772419 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:42.772898 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:42.773599 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:42.773625 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:42.774457 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:42.774464 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:42.776000 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:42.776017 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:42.776547 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:42.776554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:42.776815 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:42.777479 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:42.777506 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:42.779316 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:42.781192 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:42.781198 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:42.781213 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:42.781479 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:42.781794 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:42.781797 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:42.781799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:42.781987 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:42.782414 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:42.782422 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:42.782676 (+ 254us) client_negotiation.cc:769] Sending connection context
0807 06:22:42.782878 (+ 202us) client_negotiation.cc:240] Negotiation successful
0807 06:22:42.783324 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":118,"threads_started":1}
I20260807 06:22:42.785673 22003 heartbeater.cc:347] Connected to a master server at 127.15.254.62:37261
I20260807 06:22:42.785965 22003 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:42.786592 22003 heartbeater.cc:511] Master 127.15.254.62:37261 requested a full tablet report, sending...
I20260807 06:22:42.787858 21550 ts_manager.cc:194] Registered new tserver with Master: ee7c1849cd114aa8a5fdee67335586cf (127.15.254.2:46851)
I20260807 06:22:42.788606 21550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.2@KRBTEST.COM'} at 127.15.254.2:43723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
I20260807 06:22:42.804615 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:0
--local_ip_for_outbound_sockets=127.15.254.3
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:37261
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:46063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:42.927748 22010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:42.928005 22010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:42.928046 22010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:42.931928 22010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:42.931979 22010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:42.932034 22010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:22:42.937211 22010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:46063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:37261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:42.938323 22010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:42.939455 22010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:42.950129 22022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:42.950150 22020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:42.950263 22010 server_base.cc:1034] running on GCE node
W20260807 06:22:42.950169 22019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:42.950845 22010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:42.951579 22010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:42.952773 22010 hybrid_clock.cc:648] HybridClock initialized: now 1786083762952733 us; error 50 us; skew 500 ppm
Aug 07 06:22:42 dist-test-slave-44d2 krb5kdc[21480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083762, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:42.955780 22010 init.cc:377] Logged in from keytab as kudu/127.15.254.3@KRBTEST.COM (short username kudu)
I20260807 06:22:42.956722 22010 webserver.cc:460] Webserver started at http://127.15.254.3:39869/ using document root <none> and password file <none>
I20260807 06:22:42.957269 22010 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:42.957331 22010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:42.957510 22010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:42.959573 22010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-2/data/instance:
uuid: "cc5bd69e3f0540ab9282ab674fce1fbe"
format_stamp: "Formatted at 2026-08-07 06:22:42 on dist-test-slave-44d2"
server_key: "fac5482d747cbad201d70f756fbc11e9"
server_key_iv: "857789bdc9245baefd1a2762ff8b5019"
server_key_version: "encryptionkey@0"
I20260807 06:22:42.960101 22010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance:
uuid: "cc5bd69e3f0540ab9282ab674fce1fbe"
format_stamp: "Formatted at 2026-08-07 06:22:42 on dist-test-slave-44d2"
server_key: "fac5482d747cbad201d70f756fbc11e9"
server_key_iv: "857789bdc9245baefd1a2762ff8b5019"
server_key_version: "encryptionkey@0"
I20260807 06:22:42.964198 22010 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260807 06:22:42.966760 22033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:42.968161 22010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 06:22:42.968292 22010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "cc5bd69e3f0540ab9282ab674fce1fbe"
format_stamp: "Formatted at 2026-08-07 06:22:42 on dist-test-slave-44d2"
server_key: "fac5482d747cbad201d70f756fbc11e9"
server_key_iv: "857789bdc9245baefd1a2762ff8b5019"
server_key_version: "encryptionkey@0"
I20260807 06:22:42.968405 22010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:42.985431 22010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:42.989264 22010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:42.989500 22010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:42.990196 22010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:42.991122 22010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:42.991173 22010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:42.991245 22010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:42.991283 22010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:43.000583 22010 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:38829
I20260807 06:22:43.000610 22196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:38829 every 8 connection(s)
I20260807 06:22:43.001547 22010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:22:43.004388 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 22010
I20260807 06:22:43.004591 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance
I20260807 06:22:43.004941 16376 external_mini_cluster.cc:1428] Setting key d0ef62075e5690f82bfd255f45963bc3
Aug 07 06:22:43 dist-test-slave-44d2 krb5kdc[21480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083762, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:43.016712 21610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:43.003551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:55399 (local address 127.15.254.62:37261)
0807 06:22:43.003696 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:43.003699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:43.004709 (+ 1010us) server_negotiation.cc:407] Connection header received
0807 06:22:43.006405 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:43.006409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:43.006473 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:43.006583 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:43.007956 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:43.008578 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:43.009779 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:43.010014 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:43.010134 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:43.012845 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:43.012851 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:43.012858 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:43.012886 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:43.014447 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:43.015021 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:43.015027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:43.015032 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:43.015121 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:43.015484 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:43.015489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:43.015490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:43.015693 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:43.015809 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:43.016342 (+ 533us) server_negotiation.cc:299] Negotiation successful
0807 06:22:43.016473 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 06:22:43.017584 22199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:43.003894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:37261 (local address 127.15.254.3:55399)
0807 06:22:43.004481 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:43.004569 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:43.006175 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:43.006775 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:43.006782 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:43.007270 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:43.007805 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:43.007820 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:43.008716 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:43.008725 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:43.009640 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:43.009650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:43.010185 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:43.010189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:43.010377 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:43.011045 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:43.011063 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:43.012669 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:43.014590 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:43.014596 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:43.014613 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:43.014886 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:43.015222 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:43.015226 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:43.015228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:43.015383 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:43.015809 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:43.015817 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:43.016092 (+ 275us) client_negotiation.cc:769] Sending connection context
0807 06:22:43.016298 (+ 206us) client_negotiation.cc:240] Negotiation successful
0807 06:22:43.016678 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":194,"threads_started":1}
I20260807 06:22:43.018819 22197 heartbeater.cc:347] Connected to a master server at 127.15.254.62:37261
I20260807 06:22:43.019099 22197 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:43.019634 22197 heartbeater.cc:511] Master 127.15.254.62:37261 requested a full tablet report, sending...
I20260807 06:22:43.020758 21550 ts_manager.cc:194] Registered new tserver with Master: cc5bd69e3f0540ab9282ab674fce1fbe (127.15.254.3:38829)
I20260807 06:22:43.021317 21550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.3@KRBTEST.COM'} at 127.15.254.3:55399
I20260807 06:22:43.031318 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 06:22:43 dist-test-slave-44d2 krb5kdc[21480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 06:22:43 dist-test-slave-44d2 krb5kdc[21480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.1@KRBTEST.COM
I20260807 06:22:43.060025 22209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:43.050413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53960 (local address 127.15.254.1:38745)
0807 06:22:43.050748 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:43.050764 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:43.050789 (+ 25us) server_negotiation.cc:407] Connection header received
0807 06:22:43.050841 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:43.050846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:43.051045 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:43.051167 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:43.051833 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:43.052743 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:43.053573 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:43.053977 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:43.054067 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:43.056586 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:43.056596 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:43.056611 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:43.056636 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:43.058149 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:43.058614 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:43.058620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:43.058621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:43.058704 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:43.058996 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:43.059000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:43.059001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:43.059271 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:43.059524 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:43.059685 (+ 161us) server_negotiation.cc:299] Negotiation successful
0807 06:22:43.059849 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":118,"threads_started":1}
Aug 07 06:22:43 dist-test-slave-44d2 krb5kdc[21480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:43.071414 21610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:43.063053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52114 (local address 127.15.254.62:37261)
0807 06:22:43.063208 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:43.063212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:43.063227 (+ 15us) server_negotiation.cc:407] Connection header received
0807 06:22:43.063327 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:43.063329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:43.063373 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:43.063441 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:43.064008 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:43.064654 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:43.065437 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:43.065673 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:43.065751 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:43.068356 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:43.068360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:43.068362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:43.068390 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:43.069769 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:43.070179 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:43.070182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:43.070183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:43.070264 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:43.070499 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:43.070501 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:43.070502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:43.070745 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:43.070854 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:43.071058 (+ 204us) server_negotiation.cc:299] Negotiation successful
0807 06:22:43.071164 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260807 06:22:43.071923 21550 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52114
I20260807 06:22:43.072700 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 21638
I20260807 06:22:43.092464 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 21830
I20260807 06:22:43.109934 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 22010
I20260807 06:22:43.125488 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 21503
2026-08-07T06:22:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2470 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:22:43 dist-test-slave-44d2 krb5kdc[22216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:22:43 dist-test-slave-44d2 krb5kdc[22216](info): set up 2 sockets
Aug 07 06:22:43 dist-test-slave-44d2 krb5kdc[22216](info): commencing operation
krb5kdc: starting...
W20260807 06:22:44.350605 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:22:44 dist-test-slave-44d2 krb5kdc[22216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:22:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:22:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:22:44.491806 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:36713
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:36713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:44.610183 22232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:44.610495 22232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:44.610572 22232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:44.614789 22232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:22:44.614889 22232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:44.614923 22232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:44.614953 22232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:22:44.614988 22232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:22:44.620468 22232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:36713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:36713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:44.621716 22232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:44.622825 22232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:44.632206 22244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:44.632220 22238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:44.632534 22232 server_base.cc:1034] running on GCE node
W20260807 06:22:44.632216 22240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:44.633298 22232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:44.634547 22232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:44.635723 22232 hybrid_clock.cc:648] HybridClock initialized: now 1786083764635721 us; error 73 us; skew 500 ppm
Aug 07 06:22:44 dist-test-slave-44d2 krb5kdc[22216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083764, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:44.638789 22232 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:22:44.639554 22232 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:44.639621 22232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:44.639796 22232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:44.641983 22232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "3c4f4e793d2e4b08a9dfdf900b2df85f"
format_stamp: "Formatted at 2026-08-07 06:22:44 on dist-test-slave-44d2"
server_key: "e9be6fc37fe4b1372eb69389c7fb2623"
server_key_iv: "fb4f1ccd26f01edecd4a5e61b06f3e90"
server_key_version: "encryptionkey@0"
I20260807 06:22:44.642551 22232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "3c4f4e793d2e4b08a9dfdf900b2df85f"
format_stamp: "Formatted at 2026-08-07 06:22:44 on dist-test-slave-44d2"
server_key: "e9be6fc37fe4b1372eb69389c7fb2623"
server_key_iv: "fb4f1ccd26f01edecd4a5e61b06f3e90"
server_key_version: "encryptionkey@0"
I20260807 06:22:44.646559 22232 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:22:44.650563 22252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:44.652246 22232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:22:44.652383 22232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "3c4f4e793d2e4b08a9dfdf900b2df85f"
format_stamp: "Formatted at 2026-08-07 06:22:44 on dist-test-slave-44d2"
server_key: "e9be6fc37fe4b1372eb69389c7fb2623"
server_key_iv: "fb4f1ccd26f01edecd4a5e61b06f3e90"
server_key_version: "encryptionkey@0"
I20260807 06:22:44.652618 22232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:44.684736 22232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:44.687968 22232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:44.688205 22232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:44.695984 22232 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:36713
I20260807 06:22:44.696365 22322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:36713 every 8 connection(s)
I20260807 06:22:44.696775 22232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:22:44.699676 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 22232
I20260807 06:22:44.699824 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:22:44.699906 22323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:44.700160 16376 external_mini_cluster.cc:1428] Setting key c39445e955ce9b1d049cb9a3edd10c09
I20260807 06:22:44.707512 22323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c4f4e793d2e4b08a9dfdf900b2df85f: Bootstrap starting.
Aug 07 06:22:44 dist-test-slave-44d2 krb5kdc[22216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:44.710547 22323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c4f4e793d2e4b08a9dfdf900b2df85f: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:44.711548 22323 log.cc:824] T 00000000000000000000000000000000 P 3c4f4e793d2e4b08a9dfdf900b2df85f: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:44.714028 22323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c4f4e793d2e4b08a9dfdf900b2df85f: No bootstrap required, opened a new log
I20260807 06:22:44.715456 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:44.701645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48674 (local address 127.15.254.62:36713)
0807 06:22:44.702088 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:44.702104 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:44.702140 (+ 36us) server_negotiation.cc:407] Connection header received
0807 06:22:44.702843 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:44.702861 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:44.703209 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:44.703493 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:44.704432 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:44.705523 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:44.706760 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:44.707264 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:44.707389 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:44.710441 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:44.710447 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:44.710462 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:44.710493 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:44.712405 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:44.712973 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:44.712978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:44.712983 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:44.713131 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:44.713438 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:44.713440 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:44.713441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:44.713844 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:44.713993 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:44.714355 (+ 362us) server_negotiation.cc:299] Negotiation successful
0807 06:22:44.714699 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":155,"threads_started":1}
I20260807 06:22:44.718360 22323 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c4f4e793d2e4b08a9dfdf900b2df85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4f4e793d2e4b08a9dfdf900b2df85f" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36713 } }
I20260807 06:22:44.718621 22323 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c4f4e793d2e4b08a9dfdf900b2df85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:44.718699 22323 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c4f4e793d2e4b08a9dfdf900b2df85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4f4e793d2e4b08a9dfdf900b2df85f, State: Initialized, Role: FOLLOWER
I20260807 06:22:44.719165 22323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c4f4e793d2e4b08a9dfdf900b2df85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4f4e793d2e4b08a9dfdf900b2df85f" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36713 } }
I20260807 06:22:44.719317 22323 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c4f4e793d2e4b08a9dfdf900b2df85f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:22:44.719403 22323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c4f4e793d2e4b08a9dfdf900b2df85f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:22:44.719527 22323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c4f4e793d2e4b08a9dfdf900b2df85f [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:44.720546 22323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c4f4e793d2e4b08a9dfdf900b2df85f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4f4e793d2e4b08a9dfdf900b2df85f" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36713 } }
I20260807 06:22:44.720932 22323 leader_election.cc:302] T 00000000000000000000000000000000 P 3c4f4e793d2e4b08a9dfdf900b2df85f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c4f4e793d2e4b08a9dfdf900b2df85f; no voters:
I20260807 06:22:44.721330 22323 leader_election.cc:288] T 00000000000000000000000000000000 P 3c4f4e793d2e4b08a9dfdf900b2df85f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:22:44.721407 22331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c4f4e793d2e4b08a9dfdf900b2df85f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:44.721622 22331 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c4f4e793d2e4b08a9dfdf900b2df85f [term 1 LEADER]: Becoming Leader. State: Replica: 3c4f4e793d2e4b08a9dfdf900b2df85f, State: Running, Role: LEADER
I20260807 06:22:44.722020 22331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c4f4e793d2e4b08a9dfdf900b2df85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4f4e793d2e4b08a9dfdf900b2df85f" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36713 } }
I20260807 06:22:44.722697 22323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c4f4e793d2e4b08a9dfdf900b2df85f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:22:44.723791 22332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c4f4e793d2e4b08a9dfdf900b2df85f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c4f4e793d2e4b08a9dfdf900b2df85f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4f4e793d2e4b08a9dfdf900b2df85f" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36713 } } }
I20260807 06:22:44.723953 22332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c4f4e793d2e4b08a9dfdf900b2df85f [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:44.724267 22334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c4f4e793d2e4b08a9dfdf900b2df85f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c4f4e793d2e4b08a9dfdf900b2df85f. Latest consensus state: current_term: 1 leader_uuid: "3c4f4e793d2e4b08a9dfdf900b2df85f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4f4e793d2e4b08a9dfdf900b2df85f" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36713 } } }
I20260807 06:22:44.724399 22344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:22:44.724419 22334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c4f4e793d2e4b08a9dfdf900b2df85f [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:44.727902 22344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:22:44.733587 22344 catalog_manager.cc:1347] Generated new cluster ID: b6af9a0614ab479d96821e2074a95dcc
I20260807 06:22:44.733645 22344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:22:44.742758 22344 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:22:44.743988 22344 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:22:44.757134 22344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c4f4e793d2e4b08a9dfdf900b2df85f: Generated new TSK 0
I20260807 06:22:44.757927 22344 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
I20260807 06:22:44.793285 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:0
--local_ip_for_outbound_sockets=127.15.254.1
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:36713
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:44.914744 22358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:44.915052 22358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:44.915141 22358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:44.919731 22358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:44.919868 22358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:44.919994 22358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:22:44.926218 22358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:36713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:44.927289 22358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:44.928313 22358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:44.938418 22364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:44.938460 22365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:44.938421 22368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:44.938900 22358 server_base.cc:1034] running on GCE node
I20260807 06:22:44.939329 22358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:44.940093 22358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:44.941414 22358 hybrid_clock.cc:648] HybridClock initialized: now 1786083764941378 us; error 46 us; skew 500 ppm
Aug 07 06:22:44 dist-test-slave-44d2 krb5kdc[22216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083764, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:44.944465 22358 init.cc:377] Logged in from keytab as kudu/127.15.254.1@KRBTEST.COM (short username kudu)
I20260807 06:22:44.945461 22358 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:44.945538 22358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:44.945794 22358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:44.947978 22358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-0/data/instance:
uuid: "945d6bc828d54fd5ba1212ab6f037b48"
format_stamp: "Formatted at 2026-08-07 06:22:44 on dist-test-slave-44d2"
server_key: "dc7d8ee02a9cca0a15908fbde62c6db8"
server_key_iv: "8e0b4420e95743390e87cebf5835a18d"
server_key_version: "encryptionkey@0"
I20260807 06:22:44.948673 22358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance:
uuid: "945d6bc828d54fd5ba1212ab6f037b48"
format_stamp: "Formatted at 2026-08-07 06:22:44 on dist-test-slave-44d2"
server_key: "dc7d8ee02a9cca0a15908fbde62c6db8"
server_key_iv: "8e0b4420e95743390e87cebf5835a18d"
server_key_version: "encryptionkey@0"
I20260807 06:22:44.952723 22358 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 06:22:44.955320 22378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:44.956843 22358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:22:44.956992 22358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "945d6bc828d54fd5ba1212ab6f037b48"
format_stamp: "Formatted at 2026-08-07 06:22:44 on dist-test-slave-44d2"
server_key: "dc7d8ee02a9cca0a15908fbde62c6db8"
server_key_iv: "8e0b4420e95743390e87cebf5835a18d"
server_key_version: "encryptionkey@0"
I20260807 06:22:44.957116 22358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:44.978717 22358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:44.981827 22358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:44.981997 22358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:44.982553 22358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:44.983392 22358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:44.983444 22358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:44.983487 22358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:44.983510 22358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:44.993187 22358 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:46601
I20260807 06:22:44.993288 22548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:46601 every 8 connection(s)
I20260807 06:22:44.993871 22358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:22:45.001452 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 22358
I20260807 06:22:45.001592 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance
I20260807 06:22:45.001924 16376 external_mini_cluster.cc:1428] Setting key f657a4ca00b6e0203fbaa597cc064792
Aug 07 06:22:44 dist-test-slave-44d2 krb5kdc[22216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083764, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:45.009266 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:44.995715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:42259 (local address 127.15.254.62:36713)
0807 06:22:44.995872 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:44.995877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:44.996803 (+ 926us) server_negotiation.cc:407] Connection header received
0807 06:22:44.998130 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:44.998133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:44.998185 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:44.998267 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:44.999515 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.000085 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.001382 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.001763 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.001906 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:45.004727 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:45.004734 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:45.004741 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:45.004777 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:45.006560 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:45.007280 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:45.007284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:45.007285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:45.007384 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:45.007869 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:45.007872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:45.007873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:45.008091 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:45.008174 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:45.008993 (+ 819us) server_negotiation.cc:299] Negotiation successful
0807 06:22:45.009121 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 06:22:45.010589 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:44.996162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:36713 (local address 127.15.254.1:42259)
0807 06:22:44.996654 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:44.996699 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:44.997888 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:44.998444 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:44.998455 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:44.998854 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:44.999367 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:44.999382 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.000212 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.000215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:45.001239 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:45.001251 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.001947 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.001950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:45.002174 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:45.002819 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:45.002836 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:45.004475 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:45.006692 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:45.006699 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:45.006718 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:45.007128 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:45.007490 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:45.007496 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:45.007501 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:45.007753 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:45.008210 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:45.008220 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:45.008636 (+ 416us) client_negotiation.cc:769] Sending connection context
0807 06:22:45.008866 (+ 230us) client_negotiation.cc:240] Negotiation successful
0807 06:22:45.009315 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":122,"threads_started":1}
I20260807 06:22:45.012068 22550 heartbeater.cc:347] Connected to a master server at 127.15.254.62:36713
I20260807 06:22:45.012344 22550 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:45.012899 22550 heartbeater.cc:511] Master 127.15.254.62:36713 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.254.2@KRBTEST.COM; defaulting to no policy
I20260807 06:22:45.014635 22276 ts_manager.cc:194] Registered new tserver with Master: 945d6bc828d54fd5ba1212ab6f037b48 (127.15.254.1:46601)
I20260807 06:22:45.015911 22276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.1@KRBTEST.COM'} at 127.15.254.1:42259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
I20260807 06:22:45.035239 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:0
--local_ip_for_outbound_sockets=127.15.254.2
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:36713
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:45.152622 22555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:45.152886 22555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:45.152933 22555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:45.156934 22555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:45.156996 22555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:45.157070 22555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:22:45.162500 22555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:36713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:45.163527 22555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:45.164587 22555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:45.174448 22561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:45.174454 22565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:45.174453 22560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:45.174721 22555 server_base.cc:1034] running on GCE node
I20260807 06:22:45.175180 22555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:45.175832 22555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:45.176994 22555 hybrid_clock.cc:648] HybridClock initialized: now 1786083765176959 us; error 45 us; skew 500 ppm
Aug 07 06:22:45 dist-test-slave-44d2 krb5kdc[22216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083765, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:45.179771 22555 init.cc:377] Logged in from keytab as kudu/127.15.254.2@KRBTEST.COM (short username kudu)
I20260807 06:22:45.180621 22555 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:45.180719 22555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:45.180943 22555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:45.183014 22555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-1/data/instance:
uuid: "0565fbc1ad284703a6786b5429e7a0c0"
format_stamp: "Formatted at 2026-08-07 06:22:45 on dist-test-slave-44d2"
server_key: "994c57656d5f58f4c1504ed7e8149b09"
server_key_iv: "60b13a33a9d6d3dc962feaa88f900d3d"
server_key_version: "encryptionkey@0"
I20260807 06:22:45.183615 22555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance:
uuid: "0565fbc1ad284703a6786b5429e7a0c0"
format_stamp: "Formatted at 2026-08-07 06:22:45 on dist-test-slave-44d2"
server_key: "994c57656d5f58f4c1504ed7e8149b09"
server_key_iv: "60b13a33a9d6d3dc962feaa88f900d3d"
server_key_version: "encryptionkey@0"
I20260807 06:22:45.187458 22555 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260807 06:22:45.189960 22576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:45.191198 22555 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 06:22:45.191339 22555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "0565fbc1ad284703a6786b5429e7a0c0"
format_stamp: "Formatted at 2026-08-07 06:22:45 on dist-test-slave-44d2"
server_key: "994c57656d5f58f4c1504ed7e8149b09"
server_key_iv: "60b13a33a9d6d3dc962feaa88f900d3d"
server_key_version: "encryptionkey@0"
I20260807 06:22:45.191457 22555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:45.203953 22555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:45.207432 22555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:45.207656 22555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:45.208292 22555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:45.209218 22555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:45.209302 22555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:45.209378 22555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:45.209434 22555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:45.219030 22555 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:37801
I20260807 06:22:45.219048 22737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:37801 every 8 connection(s)
I20260807 06:22:45.219653 22555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:22:45.222493 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 22555
I20260807 06:22:45.222644 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance
I20260807 06:22:45.223006 16376 external_mini_cluster.cc:1428] Setting key b3667d4f477572deeb7a64fdc23eb123
Aug 07 06:22:45 dist-test-slave-44d2 krb5kdc[22216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083765, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.254.3@KRBTEST.COM; defaulting to no policy
I20260807 06:22:45.236229 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:45.221649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:53353 (local address 127.15.254.62:36713)
0807 06:22:45.221787 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:45.221791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:45.222730 (+ 939us) server_negotiation.cc:407] Connection header received
0807 06:22:45.224449 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:45.224453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:45.224541 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:45.224650 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:45.225898 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.226438 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.227608 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.227844 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.227961 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:45.232408 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:45.232411 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:45.232413 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:45.232439 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:45.233988 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:45.234649 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:45.234652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:45.234653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:45.234728 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:45.235152 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:45.235154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:45.235155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:45.235336 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:45.235412 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:45.235915 (+ 503us) server_negotiation.cc:299] Negotiation successful
0807 06:22:45.236063 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 06:22:45.237135 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:45.221980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:36713 (local address 127.15.254.2:53353)
0807 06:22:45.222551 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:45.222617 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:45.224241 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:45.224831 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:45.224838 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:45.225266 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:45.225754 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:45.225767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.226579 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.226582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:45.227475 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:45.227485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.228014 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.228019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:45.228191 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:45.230274 (+ 2083us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:45.230291 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:45.232218 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:45.234123 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:45.234132 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:45.234153 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:45.234542 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:45.234828 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:45.234833 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:45.234836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:45.235039 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:45.235433 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:45.235438 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:45.235699 (+ 261us) client_negotiation.cc:769] Sending connection context
0807 06:22:45.235866 (+ 167us) client_negotiation.cc:240] Negotiation successful
0807 06:22:45.236214 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.3@KRBTEST.COM" created.
I20260807 06:22:45.238467 22738 heartbeater.cc:347] Connected to a master server at 127.15.254.62:36713
I20260807 06:22:45.238746 22738 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:45.239341 22738 heartbeater.cc:511] Master 127.15.254.62:36713 requested a full tablet report, sending...
I20260807 06:22:45.240613 22276 ts_manager.cc:194] Registered new tserver with Master: 0565fbc1ad284703a6786b5429e7a0c0 (127.15.254.2:37801)
I20260807 06:22:45.241204 22276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.2@KRBTEST.COM'} at 127.15.254.2:53353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
I20260807 06:22:45.256165 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:0
--local_ip_for_outbound_sockets=127.15.254.3
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:36713
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:45.372637 22744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:45.372964 22744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:45.373059 22744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:45.377050 22744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:45.377138 22744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:45.377230 22744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:22:45.382436 22744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:36713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:45.383515 22744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:45.384581 22744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:45.394308 22750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:45.394308 22751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:45.394397 22758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:45.394807 22744 server_base.cc:1034] running on GCE node
I20260807 06:22:45.395251 22744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:45.396015 22744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:45.397205 22744 hybrid_clock.cc:648] HybridClock initialized: now 1786083765397191 us; error 47 us; skew 500 ppm
Aug 07 06:22:45 dist-test-slave-44d2 krb5kdc[22216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083765, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:45.400323 22744 init.cc:377] Logged in from keytab as kudu/127.15.254.3@KRBTEST.COM (short username kudu)
I20260807 06:22:45.401171 22744 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:45.401238 22744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:45.401407 22744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:45.403411 22744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-2/data/instance:
uuid: "ea9ce5625aae4510ac14d2fa06437541"
format_stamp: "Formatted at 2026-08-07 06:22:45 on dist-test-slave-44d2"
server_key: "8b217dd7da4c2153f1f2122c05756ac0"
server_key_iv: "b8ddf98fa711044986128da8c289c463"
server_key_version: "encryptionkey@0"
I20260807 06:22:45.403924 22744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance:
uuid: "ea9ce5625aae4510ac14d2fa06437541"
format_stamp: "Formatted at 2026-08-07 06:22:45 on dist-test-slave-44d2"
server_key: "8b217dd7da4c2153f1f2122c05756ac0"
server_key_iv: "b8ddf98fa711044986128da8c289c463"
server_key_version: "encryptionkey@0"
I20260807 06:22:45.407593 22744 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260807 06:22:45.410068 22766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:45.411324 22744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 06:22:45.411458 22744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "ea9ce5625aae4510ac14d2fa06437541"
format_stamp: "Formatted at 2026-08-07 06:22:45 on dist-test-slave-44d2"
server_key: "8b217dd7da4c2153f1f2122c05756ac0"
server_key_iv: "b8ddf98fa711044986128da8c289c463"
server_key_version: "encryptionkey@0"
I20260807 06:22:45.411568 22744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:45.430800 22744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:45.434376 22744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:45.434602 22744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:45.435264 22744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:45.436152 22744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:45.436201 22744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:45.436273 22744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:45.436311 22744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:45.446261 22744 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:33833
I20260807 06:22:45.446280 22934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:33833 every 8 connection(s)
I20260807 06:22:45.446905 22744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:22:45.454344 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 22744
I20260807 06:22:45.454486 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance
I20260807 06:22:45.454852 16376 external_mini_cluster.cc:1428] Setting key a10b57fdf0660b79dbd838062f5f40ea
Aug 07 06:22:45 dist-test-slave-44d2 krb5kdc[22216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083765, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:45.461277 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:45.448761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:59359 (local address 127.15.254.62:36713)
0807 06:22:45.448911 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:45.448915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:45.449815 (+ 900us) server_negotiation.cc:407] Connection header received
0807 06:22:45.451083 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:45.451086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:45.451134 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:45.451208 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:45.452393 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.453029 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.454176 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.454530 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.454664 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:45.457748 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:45.457751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:45.457754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:45.457779 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:45.459155 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:45.459681 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:45.459684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:45.459686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:45.459771 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:45.460135 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:45.460138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:45.460140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:45.460327 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:45.460439 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:45.460987 (+ 548us) server_negotiation.cc:299] Negotiation successful
0807 06:22:45.461114 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 06:22:45.462122 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:45.449170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:36713 (local address 127.15.254.3:59359)
0807 06:22:45.449653 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:45.449697 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:45.450873 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:45.451380 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:45.451386 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:45.451741 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:45.452253 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:45.452267 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.453166 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.453168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:45.454045 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:45.454055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.454693 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.454696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:45.454923 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:45.455679 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:45.455705 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:45.457595 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:45.459279 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:45.459285 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:45.459300 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:45.459568 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:45.459867 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:45.459870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:45.459872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:45.460018 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:45.460440 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:45.460447 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:45.460725 (+ 278us) client_negotiation.cc:769] Sending connection context
0807 06:22:45.460961 (+ 236us) client_negotiation.cc:240] Negotiation successful
0807 06:22:45.461310 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":120,"threads_started":1}
I20260807 06:22:45.463335 22935 heartbeater.cc:347] Connected to a master server at 127.15.254.62:36713
I20260807 06:22:45.463613 22935 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:45.464145 22935 heartbeater.cc:511] Master 127.15.254.62:36713 requested a full tablet report, sending...
I20260807 06:22:45.465286 22276 ts_manager.cc:194] Registered new tserver with Master: ea9ce5625aae4510ac14d2fa06437541 (127.15.254.3:33833)
I20260807 06:22:45.465878 22276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.3@KRBTEST.COM'} at 127.15.254.3:59359
I20260807 06:22:45.469077 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:22:45.478447 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:45.471089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48690 (local address 127.15.254.62:36713)
0807 06:22:45.471234 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:45.471239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:45.471369 (+ 130us) server_negotiation.cc:407] Connection header received
0807 06:22:45.471466 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:45.471468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:45.471509 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:45.471578 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:45.472211 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.472773 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.473725 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.474012 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.474105 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:45.475490 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:45.475493 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:45.475495 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:45.475516 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:45.476876 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:45.477268 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:45.477271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:45.477272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:45.477347 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:45.477587 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:45.477589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:45.477590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:45.477795 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:45.477902 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:45.478121 (+ 219us) server_negotiation.cc:299] Negotiation successful
0807 06:22:45.478219 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 06:22:45.481455 22276 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:22:45.484501 22276 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:22:45.498921 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:45.492879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:37801 (local address 127.0.0.1:54232)
0807 06:22:45.493749 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:45.493762 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:45.493833 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:45.494631 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:45.494637 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:45.494661 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:45.494807 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:45.494814 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.496172 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.496176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:45.497353 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:45.497364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.498341 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.498344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:45.498453 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:45.498490 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 06:22:45.498639 (+ 149us) client_negotiation.cc:240] Negotiation successful
0807 06:22:45.498761 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":27,"spinlock_wait_cycles":896,"thread_start_us":113,"threads_started":1}
I20260807 06:22:45.498925 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:45.493580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54232 (local address 127.15.254.2:37801)
0807 06:22:45.494087 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:45.494101 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:45.494128 (+ 27us) server_negotiation.cc:407] Connection header received
0807 06:22:45.494211 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:45.494218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:45.494436 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:45.494629 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:45.494974 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.495974 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.497484 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.498204 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.498329 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:45.498481 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:45.498672 (+ 191us) server_negotiation.cc:299] Negotiation successful
0807 06:22:45.498765 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":128,"threads_started":1}
I20260807 06:22:45.499845 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:45.493181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:46601 (local address 127.0.0.1:35822)
0807 06:22:45.494050 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:45.494076 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:45.494163 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:45.495012 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:45.495015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:45.495040 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:45.495145 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:45.495151 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.496657 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.496660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:45.498282 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:45.498293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.499432 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.499436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:45.499538 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:45.499566 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 06:22:45.499615 (+ 49us) client_negotiation.cc:240] Negotiation successful
0807 06:22:45.499700 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":192,"thread_start_us":92,"threads_started":1}
I20260807 06:22:45.500192 22950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:45.492777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:33833 (local address 127.0.0.1:47430)
0807 06:22:45.493649 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:45.493702 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:45.493833 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:45.494683 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:45.494686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:45.494717 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:45.494984 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:45.494995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.496657 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.496660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:45.498281 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:45.498293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.499684 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.499687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:45.499792 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:45.499813 (+ 21us) client_negotiation.cc:769] Sending connection context
0807 06:22:45.499864 (+ 51us) client_negotiation.cc:240] Negotiation successful
0807 06:22:45.500046 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":296,"spinlock_wait_cycles":1408,"thread_start_us":67,"threads_started":1}
I20260807 06:22:45.500644 22957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:45.493785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35822 (local address 127.15.254.1:46601)
0807 06:22:45.494538 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:45.494549 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:45.494572 (+ 23us) server_negotiation.cc:407] Connection header received
0807 06:22:45.494708 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:45.494713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:45.494885 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:45.495045 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:45.495252 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.496469 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.498434 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.499283 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.500135 (+ 852us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:45.500302 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:45.500358 (+ 56us) server_negotiation.cc:299] Negotiation successful
0807 06:22:45.500469 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":111,"threads_started":1}
I20260807 06:22:45.500983 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:45.492914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47430 (local address 127.15.254.3:33833)
0807 06:22:45.494150 (+ 1236us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:45.494162 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:45.494187 (+ 25us) server_negotiation.cc:407] Connection header received
0807 06:22:45.494269 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:45.494274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:45.494486 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:45.495131 (+ 645us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:45.495289 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.496465 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.498434 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.499536 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.500504 (+ 968us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:45.500665 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:45.500724 (+ 59us) server_negotiation.cc:299] Negotiation successful
0807 06:22:45.500822 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1115,"thread_start_us":172,"threads_started":1}
I20260807 06:22:45.502363 22648 tablet_service.cc:1467] Processing CreateTablet for tablet 9077f0bbc1244f398d25964e47ee2662 (DEFAULT_TABLE table=test-table [id=adae4acfacec44a889facbc3b9af88dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:45.503412 22648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9077f0bbc1244f398d25964e47ee2662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:45.503480 22447 tablet_service.cc:1467] Processing CreateTablet for tablet 9077f0bbc1244f398d25964e47ee2662 (DEFAULT_TABLE table=test-table [id=adae4acfacec44a889facbc3b9af88dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:45.503708 22837 tablet_service.cc:1467] Processing CreateTablet for tablet 9077f0bbc1244f398d25964e47ee2662 (DEFAULT_TABLE table=test-table [id=adae4acfacec44a889facbc3b9af88dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:45.504614 22447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9077f0bbc1244f398d25964e47ee2662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:45.504617 22837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9077f0bbc1244f398d25964e47ee2662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:45.510275 22958 tablet_bootstrap.cc:492] T 9077f0bbc1244f398d25964e47ee2662 P 0565fbc1ad284703a6786b5429e7a0c0: Bootstrap starting.
I20260807 06:22:45.511933 22960 tablet_bootstrap.cc:492] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541: Bootstrap starting.
I20260807 06:22:45.512523 22963 tablet_bootstrap.cc:492] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48: Bootstrap starting.
I20260807 06:22:45.513337 22958 tablet_bootstrap.cc:654] T 9077f0bbc1244f398d25964e47ee2662 P 0565fbc1ad284703a6786b5429e7a0c0: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:45.514328 22960 tablet_bootstrap.cc:654] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:45.514420 22963 tablet_bootstrap.cc:654] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:45.514870 22958 log.cc:824] T 9077f0bbc1244f398d25964e47ee2662 P 0565fbc1ad284703a6786b5429e7a0c0: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:45.515321 22963 log.cc:824] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:45.515645 22960 log.cc:824] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:45.517481 22958 tablet_bootstrap.cc:492] T 9077f0bbc1244f398d25964e47ee2662 P 0565fbc1ad284703a6786b5429e7a0c0: No bootstrap required, opened a new log
I20260807 06:22:45.517735 22958 ts_tablet_manager.cc:1397] T 9077f0bbc1244f398d25964e47ee2662 P 0565fbc1ad284703a6786b5429e7a0c0: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260807 06:22:45.517751 22963 tablet_bootstrap.cc:492] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48: No bootstrap required, opened a new log
I20260807 06:22:45.517751 22960 tablet_bootstrap.cc:492] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541: No bootstrap required, opened a new log
I20260807 06:22:45.517935 22960 ts_tablet_manager.cc:1397] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 06:22:45.517935 22963 ts_tablet_manager.cc:1397] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260807 06:22:45.520834 22958 raft_consensus.cc:348] T 9077f0bbc1244f398d25964e47ee2662 P 0565fbc1ad284703a6786b5429e7a0c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9ce5625aae4510ac14d2fa06437541" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33833 } } peers { permanent_uuid: "0565fbc1ad284703a6786b5429e7a0c0" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 37801 } } peers { permanent_uuid: "945d6bc828d54fd5ba1212ab6f037b48" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46601 } }
I20260807 06:22:45.521118 22958 raft_consensus.cc:374] T 9077f0bbc1244f398d25964e47ee2662 P 0565fbc1ad284703a6786b5429e7a0c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:45.521195 22958 raft_consensus.cc:729] T 9077f0bbc1244f398d25964e47ee2662 P 0565fbc1ad284703a6786b5429e7a0c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0565fbc1ad284703a6786b5429e7a0c0, State: Initialized, Role: FOLLOWER
I20260807 06:22:45.521714 22958 consensus_queue.cc:260] T 9077f0bbc1244f398d25964e47ee2662 P 0565fbc1ad284703a6786b5429e7a0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9ce5625aae4510ac14d2fa06437541" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33833 } } peers { permanent_uuid: "0565fbc1ad284703a6786b5429e7a0c0" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 37801 } } peers { permanent_uuid: "945d6bc828d54fd5ba1212ab6f037b48" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46601 } }
I20260807 06:22:45.522056 22960 raft_consensus.cc:348] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9ce5625aae4510ac14d2fa06437541" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33833 } } peers { permanent_uuid: "0565fbc1ad284703a6786b5429e7a0c0" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 37801 } } peers { permanent_uuid: "945d6bc828d54fd5ba1212ab6f037b48" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46601 } }
I20260807 06:22:45.522372 22960 raft_consensus.cc:374] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:45.522455 22960 raft_consensus.cc:729] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9ce5625aae4510ac14d2fa06437541, State: Initialized, Role: FOLLOWER
I20260807 06:22:45.522333 22963 raft_consensus.cc:348] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9ce5625aae4510ac14d2fa06437541" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33833 } } peers { permanent_uuid: "0565fbc1ad284703a6786b5429e7a0c0" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 37801 } } peers { permanent_uuid: "945d6bc828d54fd5ba1212ab6f037b48" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46601 } }
I20260807 06:22:45.522670 22963 raft_consensus.cc:374] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:45.522786 22963 raft_consensus.cc:729] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 945d6bc828d54fd5ba1212ab6f037b48, State: Initialized, Role: FOLLOWER
I20260807 06:22:45.522962 22958 ts_tablet_manager.cc:1428] T 9077f0bbc1244f398d25964e47ee2662 P 0565fbc1ad284703a6786b5429e7a0c0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260807 06:22:45.522957 22960 consensus_queue.cc:260] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9ce5625aae4510ac14d2fa06437541" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33833 } } peers { permanent_uuid: "0565fbc1ad284703a6786b5429e7a0c0" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 37801 } } peers { permanent_uuid: "945d6bc828d54fd5ba1212ab6f037b48" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46601 } }
I20260807 06:22:45.523345 22738 heartbeater.cc:503] Master 127.15.254.62:36713 was elected leader, sending a full tablet report...
I20260807 06:22:45.523806 22963 consensus_queue.cc:260] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9ce5625aae4510ac14d2fa06437541" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33833 } } peers { permanent_uuid: "0565fbc1ad284703a6786b5429e7a0c0" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 37801 } } peers { permanent_uuid: "945d6bc828d54fd5ba1212ab6f037b48" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46601 } }
I20260807 06:22:45.523922 22935 heartbeater.cc:503] Master 127.15.254.62:36713 was elected leader, sending a full tablet report...
I20260807 06:22:45.524190 22960 ts_tablet_manager.cc:1428] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260807 06:22:45.524986 22963 ts_tablet_manager.cc:1428] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260807 06:22:45.525070 22550 heartbeater.cc:503] Master 127.15.254.62:36713 was elected leader, sending a full tablet report...
W20260807 06:22:45.697850 22936 tablet.cc:2369] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 06:22:45.720768 22740 tablet.cc:2369] T 9077f0bbc1244f398d25964e47ee2662 P 0565fbc1ad284703a6786b5429e7a0c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 06:22:45.745179 22551 tablet.cc:2369] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:22:45.812307 22971 raft_consensus.cc:482] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:22:45.812620 22971 raft_consensus.cc:504] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9ce5625aae4510ac14d2fa06437541" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33833 } } peers { permanent_uuid: "0565fbc1ad284703a6786b5429e7a0c0" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 37801 } } peers { permanent_uuid: "945d6bc828d54fd5ba1212ab6f037b48" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46601 } }
I20260807 06:22:45.813889 22971 leader_election.cc:288] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea9ce5625aae4510ac14d2fa06437541 (127.15.254.3:33833), 0565fbc1ad284703a6786b5429e7a0c0 (127.15.254.2:37801)
I20260807 06:22:45.818502 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:45.814344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:43885 (local address 127.15.254.3:33833)
0807 06:22:45.814528 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:45.814533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:45.814870 (+ 337us) server_negotiation.cc:407] Connection header received
0807 06:22:45.815070 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:45.815072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:45.815116 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:45.815183 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:45.815625 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.816229 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.817451 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.817892 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.817987 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:45.818071 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:45.818277 (+ 206us) server_negotiation.cc:299] Negotiation successful
0807 06:22:45.818359 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 06:22:45.818537 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:45.814327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:39781 (local address 127.15.254.2:37801)
0807 06:22:45.814544 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:45.814549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:45.814833 (+ 284us) server_negotiation.cc:407] Connection header received
0807 06:22:45.815044 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:45.815049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:45.815245 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:45.815342 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:45.815717 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.816254 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.817494 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.817929 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.818081 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:45.818135 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:45.818348 (+ 213us) server_negotiation.cc:299] Negotiation successful
0807 06:22:45.818430 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260807 06:22:45.818533 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:45.814253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:33833 (local address 127.15.254.1:43885)
0807 06:22:45.814727 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:45.814739 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:45.814911 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:45.815248 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:45.815252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:45.815295 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:45.815501 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:45.815508 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.816356 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.816360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:45.817321 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:45.817332 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.818030 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.818033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:45.818126 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:45.818157 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 06:22:45.818232 (+ 75us) client_negotiation.cc:240] Negotiation successful
0807 06:22:45.818316 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":51,"spinlock_wait_cycles":53632,"thread_start_us":220,"threads_started":1}
I20260807 06:22:45.818539 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:45.814234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:37801 (local address 127.15.254.1:39781)
0807 06:22:45.814706 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:45.814719 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:45.814908 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:45.815379 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:45.815381 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:45.815396 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:45.815558 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:45.815583 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.816418 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.816424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:45.817325 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:45.817334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.818086 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.818093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:45.818190 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:45.818209 (+ 19us) client_negotiation.cc:769] Sending connection context
0807 06:22:45.818309 (+ 100us) client_negotiation.cc:240] Negotiation successful
0807 06:22:45.818383 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":19,"spinlock_wait_cycles":37760,"thread_start_us":242,"threads_started":1}
I20260807 06:22:45.818907 22970 raft_consensus.cc:482] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:22:45.819088 22970 raft_consensus.cc:504] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9ce5625aae4510ac14d2fa06437541" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33833 } } peers { permanent_uuid: "0565fbc1ad284703a6786b5429e7a0c0" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 37801 } } peers { permanent_uuid: "945d6bc828d54fd5ba1212ab6f037b48" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46601 } }
I20260807 06:22:45.819584 22869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9077f0bbc1244f398d25964e47ee2662" candidate_uuid: "945d6bc828d54fd5ba1212ab6f037b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea9ce5625aae4510ac14d2fa06437541" is_pre_election: true
I20260807 06:22:45.819597 22680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9077f0bbc1244f398d25964e47ee2662" candidate_uuid: "945d6bc828d54fd5ba1212ab6f037b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0565fbc1ad284703a6786b5429e7a0c0" is_pre_election: true
I20260807 06:22:45.819819 22869 raft_consensus.cc:2454] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 945d6bc828d54fd5ba1212ab6f037b48 in term 0.
I20260807 06:22:45.819890 22680 raft_consensus.cc:2454] T 9077f0bbc1244f398d25964e47ee2662 P 0565fbc1ad284703a6786b5429e7a0c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 945d6bc828d54fd5ba1212ab6f037b48 in term 0.
I20260807 06:22:45.820379 22379 leader_election.cc:302] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 945d6bc828d54fd5ba1212ab6f037b48, ea9ce5625aae4510ac14d2fa06437541; no voters:
I20260807 06:22:45.820442 22970 leader_election.cc:288] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0565fbc1ad284703a6786b5429e7a0c0 (127.15.254.2:37801), 945d6bc828d54fd5ba1212ab6f037b48 (127.15.254.1:46601)
I20260807 06:22:45.820848 22971 raft_consensus.cc:2781] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:22:45.820988 22971 raft_consensus.cc:482] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:22:45.821054 22971 raft_consensus.cc:3037] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:45.822527 22971 raft_consensus.cc:504] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9ce5625aae4510ac14d2fa06437541" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33833 } } peers { permanent_uuid: "0565fbc1ad284703a6786b5429e7a0c0" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 37801 } } peers { permanent_uuid: "945d6bc828d54fd5ba1212ab6f037b48" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46601 } }
I20260807 06:22:45.823294 22869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9077f0bbc1244f398d25964e47ee2662" candidate_uuid: "945d6bc828d54fd5ba1212ab6f037b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea9ce5625aae4510ac14d2fa06437541"
I20260807 06:22:45.823352 22680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9077f0bbc1244f398d25964e47ee2662" candidate_uuid: "945d6bc828d54fd5ba1212ab6f037b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0565fbc1ad284703a6786b5429e7a0c0"
I20260807 06:22:45.823436 22869 raft_consensus.cc:3037] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:45.823488 22680 raft_consensus.cc:3037] T 9077f0bbc1244f398d25964e47ee2662 P 0565fbc1ad284703a6786b5429e7a0c0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:45.824851 22869 raft_consensus.cc:2454] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 945d6bc828d54fd5ba1212ab6f037b48 in term 1.
I20260807 06:22:45.824877 22680 raft_consensus.cc:2454] T 9077f0bbc1244f398d25964e47ee2662 P 0565fbc1ad284703a6786b5429e7a0c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 945d6bc828d54fd5ba1212ab6f037b48 in term 1.
I20260807 06:22:45.825223 22382 leader_election.cc:302] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0565fbc1ad284703a6786b5429e7a0c0, 945d6bc828d54fd5ba1212ab6f037b48; no voters:
I20260807 06:22:45.825547 22971 leader_election.cc:288] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48 [CANDIDATE]: Term 1 election: Requested vote from peers ea9ce5625aae4510ac14d2fa06437541 (127.15.254.3:33833), 0565fbc1ad284703a6786b5429e7a0c0 (127.15.254.2:37801)
I20260807 06:22:45.825635 22977 raft_consensus.cc:2781] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:45.825796 22977 raft_consensus.cc:686] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48 [term 1 LEADER]: Becoming Leader. State: Replica: 945d6bc828d54fd5ba1212ab6f037b48, State: Running, Role: LEADER
I20260807 06:22:45.825838 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:45.820690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:37801 (local address 127.15.254.3:38199)
0807 06:22:45.820863 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:45.820879 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:45.820962 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:45.821748 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:45.821751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:45.821784 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:45.821884 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:45.821888 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.822929 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.822933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:45.824235 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:45.824249 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.825328 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.825332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:45.825478 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:45.825508 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 06:22:45.825598 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 06:22:45.825692 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":4,"spinlock_wait_cycles":1280}
I20260807 06:22:45.825820 22957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:45.820779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:38079 (local address 127.15.254.1:46601)
0807 06:22:45.820885 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:45.820890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:45.821456 (+ 566us) server_negotiation.cc:407] Connection header received
0807 06:22:45.821559 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:45.821562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:45.821613 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:45.821681 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:45.821987 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.822749 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.824425 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.825055 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.825142 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:45.825196 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:45.825614 (+ 418us) server_negotiation.cc:299] Negotiation successful
0807 06:22:45.825687 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260807 06:22:45.826272 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:45.821290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:38199 (local address 127.15.254.2:37801)
0807 06:22:45.821435 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:45.821441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:45.821456 (+ 15us) server_negotiation.cc:407] Connection header received
0807 06:22:45.821498 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:45.821501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:45.821558 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:45.821626 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:45.822029 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.822808 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.824465 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.825100 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.825215 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:45.825272 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:45.826090 (+ 818us) server_negotiation.cc:299] Negotiation successful
0807 06:22:45.826164 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 06:22:45.826287 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:45.820690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:46601 (local address 127.15.254.3:38079)
0807 06:22:45.821333 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:45.821348 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:45.821470 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:45.821709 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:45.821712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:45.821729 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:45.821867 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:45.821873 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.822872 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.822879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:45.824269 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:45.824281 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.825296 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.825299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:45.825452 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:45.825481 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 06:22:45.825871 (+ 390us) client_negotiation.cc:240] Negotiation successful
0807 06:22:45.826148 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":265,"thread_start_us":146,"threads_started":1}
I20260807 06:22:45.826303 22977 consensus_queue.cc:237] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9ce5625aae4510ac14d2fa06437541" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33833 } } peers { permanent_uuid: "0565fbc1ad284703a6786b5429e7a0c0" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 37801 } } peers { permanent_uuid: "945d6bc828d54fd5ba1212ab6f037b48" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46601 } }
I20260807 06:22:45.826803 22680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9077f0bbc1244f398d25964e47ee2662" candidate_uuid: "ea9ce5625aae4510ac14d2fa06437541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0565fbc1ad284703a6786b5429e7a0c0" is_pre_election: true
I20260807 06:22:45.826850 22479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9077f0bbc1244f398d25964e47ee2662" candidate_uuid: "ea9ce5625aae4510ac14d2fa06437541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "945d6bc828d54fd5ba1212ab6f037b48" is_pre_election: true
I20260807 06:22:45.827035 22680 raft_consensus.cc:2379] T 9077f0bbc1244f398d25964e47ee2662 P 0565fbc1ad284703a6786b5429e7a0c0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea9ce5625aae4510ac14d2fa06437541 in current term 1: Already voted for candidate 945d6bc828d54fd5ba1212ab6f037b48 in this term.
I20260807 06:22:45.827647 22774 leader_election.cc:302] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea9ce5625aae4510ac14d2fa06437541; no voters: 0565fbc1ad284703a6786b5429e7a0c0, 945d6bc828d54fd5ba1212ab6f037b48
I20260807 06:22:45.827919 22970 raft_consensus.cc:2726] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 06:22:45.830374 22273 catalog_manager.cc:5495] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 945d6bc828d54fd5ba1212ab6f037b48 (127.15.254.1). New cstate: current_term: 1 leader_uuid: "945d6bc828d54fd5ba1212ab6f037b48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9ce5625aae4510ac14d2fa06437541" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0565fbc1ad284703a6786b5429e7a0c0" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 37801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "945d6bc828d54fd5ba1212ab6f037b48" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46601 } health_report { overall_health: HEALTHY } } }
I20260807 06:22:45.864300 22957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:45.860429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35830 (local address 127.15.254.1:46601)
0807 06:22:45.860606 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:45.860611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:45.860644 (+ 33us) server_negotiation.cc:407] Connection header received
0807 06:22:45.860855 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:45.860859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:45.860933 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:45.861041 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:22:45.861326 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.861933 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.862773 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:45.862985 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:45.863063 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:45.863418 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:22:45.863866 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:22:45.863986 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:45.864041 (+ 55us) server_negotiation.cc:299] Negotiation successful
0807 06:22:45.864095 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 06:22:45.871228 22680 raft_consensus.cc:1261] T 9077f0bbc1244f398d25964e47ee2662 P 0565fbc1ad284703a6786b5429e7a0c0 [term 1 FOLLOWER]: Refusing update from remote peer 945d6bc828d54fd5ba1212ab6f037b48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:22:45.871266 22869 raft_consensus.cc:1261] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541 [term 1 FOLLOWER]: Refusing update from remote peer 945d6bc828d54fd5ba1212ab6f037b48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:22:45.871997 22977 consensus_queue.cc:1035] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0565fbc1ad284703a6786b5429e7a0c0" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 37801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:22:45.872239 22971 consensus_queue.cc:1035] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea9ce5625aae4510ac14d2fa06437541" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:22:45.879858 22983 mvcc.cc:205] Tried to move back new op lower bound from 7315799104998678528 to 7315799104827953152. Current Snapshot: MvccSnapshot[applied={T|T < 7315799104998678528}]
I20260807 06:22:45.883738 22984 mvcc.cc:205] Tried to move back new op lower bound from 7315799104998678528 to 7315799104827953152. Current Snapshot: MvccSnapshot[applied={T|T < 7315799104998678528 or (T in {7315799104998678528})}]
I20260807 06:22:45.889278 22274 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:22:45.889433 22274 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:22:45.891852 22274 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c4f4e793d2e4b08a9dfdf900b2df85f: Sending DeleteTablet for 3 replicas of tablet 9077f0bbc1244f398d25964e47ee2662
I20260807 06:22:45.892728 22648 tablet_service.cc:1514] Processing DeleteTablet for tablet 9077f0bbc1244f398d25964e47ee2662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:45 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:54232
I20260807 06:22:45.893129 22447 tablet_service.cc:1514] Processing DeleteTablet for tablet 9077f0bbc1244f398d25964e47ee2662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:45 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:35822
I20260807 06:22:45.893303 22837 tablet_service.cc:1514] Processing DeleteTablet for tablet 9077f0bbc1244f398d25964e47ee2662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:45 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:47430
I20260807 06:22:45.893635 22995 tablet_replica.cc:330] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48: stopping tablet replica
I20260807 06:22:45.893899 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 22358
I20260807 06:22:45.893958 22995 raft_consensus.cc:2229] T 9077f0bbc1244f398d25964e47ee2662 P 945d6bc828d54fd5ba1212ab6f037b48 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:22:45.900164 22996 tablet_replica.cc:330] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541: stopping tablet replica
I20260807 06:22:45.900164 22994 tablet_replica.cc:330] T 9077f0bbc1244f398d25964e47ee2662 P 0565fbc1ad284703a6786b5429e7a0c0: stopping tablet replica
I20260807 06:22:45.900574 22996 raft_consensus.cc:2229] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:22:45.900575 22994 raft_consensus.cc:2229] T 9077f0bbc1244f398d25964e47ee2662 P 0565fbc1ad284703a6786b5429e7a0c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:22:45.900955 22994 raft_consensus.cc:2258] T 9077f0bbc1244f398d25964e47ee2662 P 0565fbc1ad284703a6786b5429e7a0c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:45.900955 22996 raft_consensus.cc:2258] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:45.902686 22994 ts_tablet_manager.cc:1905] T 9077f0bbc1244f398d25964e47ee2662 P 0565fbc1ad284703a6786b5429e7a0c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:22:45.902686 22996 ts_tablet_manager.cc:1905] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:22:45.905803 22994 ts_tablet_manager.cc:1918] T 9077f0bbc1244f398d25964e47ee2662 P 0565fbc1ad284703a6786b5429e7a0c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:22:45.905929 22994 log.cc:1196] T 9077f0bbc1244f398d25964e47ee2662 P 0565fbc1ad284703a6786b5429e7a0c0: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/9077f0bbc1244f398d25964e47ee2662
I20260807 06:22:45.906049 22996 ts_tablet_manager.cc:1918] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:22:45.906159 22996 log.cc:1196] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/9077f0bbc1244f398d25964e47ee2662
I20260807 06:22:45.906383 22994 ts_tablet_manager.cc:1939] T 9077f0bbc1244f398d25964e47ee2662 P 0565fbc1ad284703a6786b5429e7a0c0: Deleting consensus metadata
I20260807 06:22:45.906543 22996 ts_tablet_manager.cc:1939] T 9077f0bbc1244f398d25964e47ee2662 P ea9ce5625aae4510ac14d2fa06437541: Deleting consensus metadata
I20260807 06:22:45.907791 22254 catalog_manager.cc:4841] TS ea9ce5625aae4510ac14d2fa06437541 (127.15.254.3:33833): tablet 9077f0bbc1244f398d25964e47ee2662 (table test-table [id=adae4acfacec44a889facbc3b9af88dc]) successfully deleted
W20260807 06:22:45.907864 22256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.254.1:46601 (error 108)
I20260807 06:22:45.908368 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 22555
I20260807 06:22:45.908456 22258 catalog_manager.cc:4841] TS 0565fbc1ad284703a6786b5429e7a0c0 (127.15.254.2:37801): tablet 9077f0bbc1244f398d25964e47ee2662 (table test-table [id=adae4acfacec44a889facbc3b9af88dc]) successfully deleted
I20260807 06:22:45.911604 22950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:45.911270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:46601 (local address 127.0.0.1:35832)
0807 06:22:45.911426 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:45.911505 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.1:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 06:22:45.912402 22256 catalog_manager.cc:4584] TS 945d6bc828d54fd5ba1212ab6f037b48 (127.15.254.1:46601): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9077f0bbc1244f398d25964e47ee2662: Network error: Client connection negotiation failed: client connection to 127.15.254.1:46601: connect: Connection refused (error 111)
I20260807 06:22:45.921344 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 22744
I20260807 06:22:45.935010 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 22232
2026-08-07T06:22:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2813 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-07T06:22:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:22:45Z Disabled control of system clock
I20260807 06:22:45.976233 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:37435
--webserver_interface=127.15.254.62
--webserver_port=0
--builtin_ntp_servers=127.15.254.20:44993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:37435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 06:22:46.091904 23001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:46.092182 23001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:46.092222 23001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:46.096210 23001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:22:46.096275 23001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:46.096292 23001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:46.096310 23001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:22:46.096325 23001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:22:46.100994 23001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:44993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:37435
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:37435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:46.102125 23001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:46.103492 23001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:46.112336 23013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:46.112442 23011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:46.112285 23009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:46.112759 23001 server_base.cc:1034] running on GCE node
I20260807 06:22:46.113395 23001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:46.114531 23001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:46.115737 23001 hybrid_clock.cc:648] HybridClock initialized: now 1786083766115717 us; error 30 us; skew 500 ppm
I20260807 06:22:46.121433 23001 webserver.cc:460] Webserver started at http://127.15.254.62:46367/ using document root <none> and password file <none>
I20260807 06:22:46.122100 23001 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:46.122202 23001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:46.122453 23001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:46.124683 23001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "dcb225ce72c94d9fabc68f31ded5081b"
format_stamp: "Formatted at 2026-08-07 06:22:46 on dist-test-slave-44d2"
server_key: "2c6f7269506d447db1afd8de5ff4a248"
server_key_iv: "762d52ab656de3809a7ab26ef3c3a0bd"
server_key_version: "encryptionkey@0"
I20260807 06:22:46.125288 23001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "dcb225ce72c94d9fabc68f31ded5081b"
format_stamp: "Formatted at 2026-08-07 06:22:46 on dist-test-slave-44d2"
server_key: "2c6f7269506d447db1afd8de5ff4a248"
server_key_iv: "762d52ab656de3809a7ab26ef3c3a0bd"
server_key_version: "encryptionkey@0"
I20260807 06:22:46.129046 23001 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260807 06:22:46.131632 23021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:46.133093 23001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:22:46.133244 23001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "dcb225ce72c94d9fabc68f31ded5081b"
format_stamp: "Formatted at 2026-08-07 06:22:46 on dist-test-slave-44d2"
server_key: "2c6f7269506d447db1afd8de5ff4a248"
server_key_iv: "762d52ab656de3809a7ab26ef3c3a0bd"
server_key_version: "encryptionkey@0"
I20260807 06:22:46.133361 23001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:46.137988 23001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:46.138623 23001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:46.138813 23001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:46.146034 23001 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:37435
I20260807 06:22:46.146071 23099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:37435 every 8 connection(s)
I20260807 06:22:46.146972 23001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:22:46.150525 23101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:46.152787 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 23001
I20260807 06:22:46.152946 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:22:46.153241 16376 external_mini_cluster.cc:1428] Setting key 064558437a476e579b85f2f475de8862
I20260807 06:22:46.156833 23101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcb225ce72c94d9fabc68f31ded5081b: Bootstrap starting.
I20260807 06:22:46.159075 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:46.154753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51156 (local address 127.15.254.62:37435)
0807 06:22:46.155416 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:46.155435 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:46.155471 (+ 36us) server_negotiation.cc:407] Connection header received
0807 06:22:46.156128 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:46.156146 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:46.156565 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:46.156854 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:46.157748 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:46.157753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:46.157765 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:22:46.157775 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:46.157835 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:22:46.157851 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:46.157958 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:46.158102 (+ 144us) server_negotiation.cc:299] Negotiation successful
0807 06:22:46.158367 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":143,"threads_started":1}
I20260807 06:22:46.159482 23101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcb225ce72c94d9fabc68f31ded5081b: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:46.160245 23101 log.cc:824] T 00000000000000000000000000000000 P dcb225ce72c94d9fabc68f31ded5081b: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:46.162252 23101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcb225ce72c94d9fabc68f31ded5081b: No bootstrap required, opened a new log
I20260807 06:22:46.166157 23101 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcb225ce72c94d9fabc68f31ded5081b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb225ce72c94d9fabc68f31ded5081b" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37435 } }
I20260807 06:22:46.166407 23101 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcb225ce72c94d9fabc68f31ded5081b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:46.166494 23101 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcb225ce72c94d9fabc68f31ded5081b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcb225ce72c94d9fabc68f31ded5081b, State: Initialized, Role: FOLLOWER
I20260807 06:22:46.166916 23101 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcb225ce72c94d9fabc68f31ded5081b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb225ce72c94d9fabc68f31ded5081b" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37435 } }
I20260807 06:22:46.167073 23101 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcb225ce72c94d9fabc68f31ded5081b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:22:46.167161 23101 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcb225ce72c94d9fabc68f31ded5081b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:22:46.167287 23101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcb225ce72c94d9fabc68f31ded5081b [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:46.168172 23101 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcb225ce72c94d9fabc68f31ded5081b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb225ce72c94d9fabc68f31ded5081b" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37435 } }
I20260807 06:22:46.168563 23101 leader_election.cc:302] T 00000000000000000000000000000000 P dcb225ce72c94d9fabc68f31ded5081b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcb225ce72c94d9fabc68f31ded5081b; no voters:
I20260807 06:22:46.168885 23101 leader_election.cc:288] T 00000000000000000000000000000000 P dcb225ce72c94d9fabc68f31ded5081b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:22:46.168964 23107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcb225ce72c94d9fabc68f31ded5081b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:46.169277 23107 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcb225ce72c94d9fabc68f31ded5081b [term 1 LEADER]: Becoming Leader. State: Replica: dcb225ce72c94d9fabc68f31ded5081b, State: Running, Role: LEADER
I20260807 06:22:46.169734 23107 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcb225ce72c94d9fabc68f31ded5081b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb225ce72c94d9fabc68f31ded5081b" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37435 } }
I20260807 06:22:46.170171 23101 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcb225ce72c94d9fabc68f31ded5081b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:22:46.171628 23109 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcb225ce72c94d9fabc68f31ded5081b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcb225ce72c94d9fabc68f31ded5081b. Latest consensus state: current_term: 1 leader_uuid: "dcb225ce72c94d9fabc68f31ded5081b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb225ce72c94d9fabc68f31ded5081b" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37435 } } }
I20260807 06:22:46.171669 23108 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcb225ce72c94d9fabc68f31ded5081b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcb225ce72c94d9fabc68f31ded5081b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb225ce72c94d9fabc68f31ded5081b" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37435 } } }
I20260807 06:22:46.171813 23108 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcb225ce72c94d9fabc68f31ded5081b [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:46.172092 23109 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcb225ce72c94d9fabc68f31ded5081b [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:46.173074 23119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:22:46.175971 23119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:22:46.181208 23119 catalog_manager.cc:1347] Generated new cluster ID: 07c4aa69c6c64052b114ad1927cb048a
I20260807 06:22:46.181277 23119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:22:46.204123 23119 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:22:46.204551 23119 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:22:46.211266 23119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcb225ce72c94d9fabc68f31ded5081b: Generated new TSK 0
I20260807 06:22:46.212138 23119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:22:46.223213 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:0
--local_ip_for_outbound_sockets=127.15.254.1
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:37435
--builtin_ntp_servers=127.15.254.20:44993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 06:22:46.346194 23131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:46.346477 23131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:46.346520 23131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:46.350673 23131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:46.350731 23131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:46.350788 23131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:22:46.355712 23131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:44993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:37435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:46.356843 23131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:46.358214 23131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:46.368544 23138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:46.368558 23140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:46.368558 23142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:46.369112 23131 server_base.cc:1034] running on GCE node
I20260807 06:22:46.369554 23131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:46.370342 23131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:46.371563 23131 hybrid_clock.cc:648] HybridClock initialized: now 1786083766371516 us; error 60 us; skew 500 ppm
I20260807 06:22:46.373690 23131 webserver.cc:460] Webserver started at http://127.15.254.1:37877/ using document root <none> and password file <none>
I20260807 06:22:46.374341 23131 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:46.374444 23131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:46.374696 23131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:46.377220 23131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-0/data/instance:
uuid: "895a15a8509e47afb008de2720b672c5"
format_stamp: "Formatted at 2026-08-07 06:22:46 on dist-test-slave-44d2"
server_key: "17d6bd91d894d5f7ec41b609dad060b5"
server_key_iv: "8e00b26b9a561738e9cc8140e7f51215"
server_key_version: "encryptionkey@0"
I20260807 06:22:46.377830 23131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance:
uuid: "895a15a8509e47afb008de2720b672c5"
format_stamp: "Formatted at 2026-08-07 06:22:46 on dist-test-slave-44d2"
server_key: "17d6bd91d894d5f7ec41b609dad060b5"
server_key_iv: "8e00b26b9a561738e9cc8140e7f51215"
server_key_version: "encryptionkey@0"
I20260807 06:22:46.381997 23131 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:22:46.384862 23150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:46.386297 23131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:22:46.386453 23131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "895a15a8509e47afb008de2720b672c5"
format_stamp: "Formatted at 2026-08-07 06:22:46 on dist-test-slave-44d2"
server_key: "17d6bd91d894d5f7ec41b609dad060b5"
server_key_iv: "8e00b26b9a561738e9cc8140e7f51215"
server_key_version: "encryptionkey@0"
I20260807 06:22:46.386629 23131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:46.391678 23131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:46.392341 23131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:46.392611 23131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:46.393229 23131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:46.394295 23131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:46.394374 23131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:46.394438 23131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:46.394486 23131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260807 06:22:46.404696 23131 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:42955
I20260807 06:22:46.404722 23318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:42955 every 8 connection(s)
I20260807 06:22:46.405670 23131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:22:46.411087 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 23131
I20260807 06:22:46.411230 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance
I20260807 06:22:46.411518 16376 external_mini_cluster.cc:1428] Setting key 3dfc97bbf2beffddc66b9c23f0fa4a9f
I20260807 06:22:46.412287 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:46.407631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:44805 (local address 127.15.254.62:37435)
0807 06:22:46.407768 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:46.407772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:46.408899 (+ 1127us) server_negotiation.cc:407] Connection header received
0807 06:22:46.410358 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:46.410361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:46.410411 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:46.410525 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:46.411783 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:46.411787 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:46.411790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:22:46.411810 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:46.411831 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:22:46.411840 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:46.411942 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:46.412100 (+ 158us) server_negotiation.cc:299] Negotiation successful
0807 06:22:46.412150 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 06:22:46.413273 23321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:46.407966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:37435 (local address 127.15.254.1:44805)
0807 06:22:46.408748 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:46.408790 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:46.410133 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:46.410699 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:46.410706 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:46.410794 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:46.411467 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:22:46.411469 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:46.411515 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:22:46.411584 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:46.411924 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:46.411934 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:46.411940 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 06:22:46.412136 (+ 196us) client_negotiation.cc:240] Negotiation successful
0807 06:22:46.412447 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":119,"threads_started":1}
I20260807 06:22:46.413818 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:0
--local_ip_for_outbound_sockets=127.15.254.2
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:37435
--builtin_ntp_servers=127.15.254.20:44993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260807 06:22:46.414569 23319 heartbeater.cc:347] Connected to a master server at 127.15.254.62:37435
I20260807 06:22:46.414805 23319 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:46.415217 23319 heartbeater.cc:511] Master 127.15.254.62:37435 requested a full tablet report, sending...
I20260807 06:22:46.416800 23047 ts_manager.cc:194] Registered new tserver with Master: 895a15a8509e47afb008de2720b672c5 (127.15.254.1:42955)
W20260807 06:22:46.534258 23323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:46.534643 23323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:46.534736 23323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:46.538857 23323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:46.538935 23323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:46.539021 23323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:22:46.543794 23323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:44993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:37435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:46.544966 23323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:46.546473 23323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:46.556787 23336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:46.556769 23333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:46.556769 23340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:46.556823 23323 server_base.cc:1034] running on GCE node
I20260807 06:22:46.557469 23323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:46.558189 23323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:46.559386 23323 hybrid_clock.cc:648] HybridClock initialized: now 1786083766559356 us; error 50 us; skew 500 ppm
I20260807 06:22:46.561476 23323 webserver.cc:460] Webserver started at http://127.15.254.2:46235/ using document root <none> and password file <none>
I20260807 06:22:46.562099 23323 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:46.562196 23323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:46.562444 23323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:46.564648 23323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-1/data/instance:
uuid: "439e5263aaea4e71928abba3dfa8be0c"
format_stamp: "Formatted at 2026-08-07 06:22:46 on dist-test-slave-44d2"
server_key: "ab75bb7884fa58c2b9047b01153e7558"
server_key_iv: "96cf6f6790657aeae728319f3a29d27f"
server_key_version: "encryptionkey@0"
I20260807 06:22:46.565234 23323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance:
uuid: "439e5263aaea4e71928abba3dfa8be0c"
format_stamp: "Formatted at 2026-08-07 06:22:46 on dist-test-slave-44d2"
server_key: "ab75bb7884fa58c2b9047b01153e7558"
server_key_iv: "96cf6f6790657aeae728319f3a29d27f"
server_key_version: "encryptionkey@0"
I20260807 06:22:46.569139 23323 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 06:22:46.571967 23348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:46.573379 23323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:22:46.573513 23323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "439e5263aaea4e71928abba3dfa8be0c"
format_stamp: "Formatted at 2026-08-07 06:22:46 on dist-test-slave-44d2"
server_key: "ab75bb7884fa58c2b9047b01153e7558"
server_key_iv: "96cf6f6790657aeae728319f3a29d27f"
server_key_version: "encryptionkey@0"
I20260807 06:22:46.573625 23323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:46.578454 23323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:46.579092 23323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:46.579278 23323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:46.579854 23323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:46.580854 23323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:46.580904 23323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:46.580969 23323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:46.581007 23323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:46.589771 23323 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:42723
I20260807 06:22:46.589859 23505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:42723 every 8 connection(s)
I20260807 06:22:46.590751 23323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:22:46.591766 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 23323
I20260807 06:22:46.591905 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance
I20260807 06:22:46.592191 16376 external_mini_cluster.cc:1428] Setting key 815f9152aed072e8932e512b3f145f72
I20260807 06:22:46.594306 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:0
--local_ip_for_outbound_sockets=127.15.254.3
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:37435
--builtin_ntp_servers=127.15.254.20:44993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260807 06:22:46.598004 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:46.592901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:42379 (local address 127.15.254.62:37435)
0807 06:22:46.593039 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:46.593044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:46.594393 (+ 1349us) server_negotiation.cc:407] Connection header received
0807 06:22:46.595788 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:46.595795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:46.595851 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:46.595961 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:46.597430 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:46.597434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:46.597435 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:22:46.597448 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:46.597471 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:22:46.597479 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:46.597560 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:46.597761 (+ 201us) server_negotiation.cc:299] Negotiation successful
0807 06:22:46.597816 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 06:22:46.598860 23509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:46.593361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:37435 (local address 127.15.254.2:42379)
0807 06:22:46.594217 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:46.594271 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:46.595560 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:46.596283 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:46.596293 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:46.596420 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:46.597086 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:22:46.597089 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:46.597152 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:22:46.597250 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:46.597575 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:46.597587 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:46.597594 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 06:22:46.597781 (+ 187us) client_negotiation.cc:240] Negotiation successful
0807 06:22:46.598032 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":176,"threads_started":1}
I20260807 06:22:46.600201 23506 heartbeater.cc:347] Connected to a master server at 127.15.254.62:37435
I20260807 06:22:46.600447 23506 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:46.600935 23506 heartbeater.cc:511] Master 127.15.254.62:37435 requested a full tablet report, sending...
I20260807 06:22:46.602110 23047 ts_manager.cc:194] Registered new tserver with Master: 439e5263aaea4e71928abba3dfa8be0c (127.15.254.2:42723)
W20260807 06:22:46.718604 23510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:46.718868 23510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:46.718909 23510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:46.723405 23510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:46.723484 23510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:46.723589 23510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:22:46.728732 23510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:44993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:37435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:46.729923 23510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:46.731371 23510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 06:22:46.742170 23510 server_base.cc:1034] running on GCE node
W20260807 06:22:46.742151 23518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:46.742120 23517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:46.742120 23522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:46.742782 23510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:46.743384 23510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:46.744685 23510 hybrid_clock.cc:648] HybridClock initialized: now 1786083766744650 us; error 52 us; skew 500 ppm
I20260807 06:22:46.746595 23510 webserver.cc:460] Webserver started at http://127.15.254.3:40303/ using document root <none> and password file <none>
I20260807 06:22:46.747217 23510 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:46.747283 23510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:46.747556 23510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:46.749774 23510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-2/data/instance:
uuid: "0933341a5d7641cf9ed58d718739e435"
format_stamp: "Formatted at 2026-08-07 06:22:46 on dist-test-slave-44d2"
server_key: "4cd284c2a02ead4f1fdce1e8148ea7f5"
server_key_iv: "f359663ff17b8d20f73f9ccb649996a9"
server_key_version: "encryptionkey@0"
I20260807 06:22:46.750381 23510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance:
uuid: "0933341a5d7641cf9ed58d718739e435"
format_stamp: "Formatted at 2026-08-07 06:22:46 on dist-test-slave-44d2"
server_key: "4cd284c2a02ead4f1fdce1e8148ea7f5"
server_key_iv: "f359663ff17b8d20f73f9ccb649996a9"
server_key_version: "encryptionkey@0"
I20260807 06:22:46.754380 23510 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:22:46.757107 23530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:46.758389 23510 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 06:22:46.758523 23510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "0933341a5d7641cf9ed58d718739e435"
format_stamp: "Formatted at 2026-08-07 06:22:46 on dist-test-slave-44d2"
server_key: "4cd284c2a02ead4f1fdce1e8148ea7f5"
server_key_iv: "f359663ff17b8d20f73f9ccb649996a9"
server_key_version: "encryptionkey@0"
I20260807 06:22:46.758641 23510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:46.763427 23510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:46.764060 23510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:46.764252 23510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:46.764878 23510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:46.765861 23510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:46.765910 23510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:46.765978 23510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:46.766036 23510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:46.775803 23510 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:34049
I20260807 06:22:46.775825 23700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:34049 every 8 connection(s)
I20260807 06:22:46.776866 23510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:22:46.782680 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 23510
I20260807 06:22:46.782829 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance
I20260807 06:22:46.783131 16376 external_mini_cluster.cc:1428] Setting key 66f8aee88a04876535f6cbc23ea48ddf
I20260807 06:22:46.783191 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:46.778761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:59055 (local address 127.15.254.62:37435)
0807 06:22:46.778938 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:46.778943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:46.780028 (+ 1085us) server_negotiation.cc:407] Connection header received
0807 06:22:46.781396 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:46.781415 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:46.781456 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:46.781533 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:46.782611 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:46.782616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:46.782618 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:22:46.782635 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:46.782661 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:22:46.782674 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:46.782783 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:46.782987 (+ 204us) server_negotiation.cc:299] Negotiation successful
0807 06:22:46.783049 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 06:22:46.784216 23705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:46.779204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:37435 (local address 127.15.254.3:59055)
0807 06:22:46.779869 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:46.779909 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:46.781192 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:46.781718 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:46.781725 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:46.781813 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:46.782331 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:22:46.782332 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:46.782373 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:22:46.782442 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:46.782795 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:46.782807 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:46.782815 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 06:22:46.783017 (+ 202us) client_negotiation.cc:240] Negotiation successful
0807 06:22:46.783265 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":107,"threads_started":1}
I20260807 06:22:46.785532 23701 heartbeater.cc:347] Connected to a master server at 127.15.254.62:37435
I20260807 06:22:46.785734 23701 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:46.786092 23701 heartbeater.cc:511] Master 127.15.254.62:37435 requested a full tablet report, sending...
I20260807 06:22:46.787077 23047 ts_manager.cc:194] Registered new tserver with Master: 0933341a5d7641cf9ed58d718739e435 (127.15.254.3:34049)
I20260807 06:22:46.797469 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:22:46.801182 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:46.799478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51170 (local address 127.15.254.62:37435)
0807 06:22:46.799626 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:46.799632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:46.799833 (+ 201us) server_negotiation.cc:407] Connection header received
0807 06:22:46.799925 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:46.799928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:46.799963 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:46.800057 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:46.800716 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:46.800719 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:46.800721 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:22:46.800732 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:46.800751 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:22:46.800759 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:46.800828 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:46.800898 (+ 70us) server_negotiation.cc:299] Negotiation successful
0807 06:22:46.800954 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 06:22:46.804019 23047 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:22:46.807132 23047 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:22:46.819798 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:46.816060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36130 (local address 127.15.254.1:42955)
0807 06:22:46.816413 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:46.816426 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:46.817011 (+ 585us) server_negotiation.cc:407] Connection header received
0807 06:22:46.817464 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:46.817471 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:46.817666 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:46.817788 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:46.819128 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:46.819133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:46.819143 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:22:46.819155 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:46.819194 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:22:46.819230 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:46.819332 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:46.819517 (+ 185us) server_negotiation.cc:299] Negotiation successful
0807 06:22:46.819615 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":150,"threads_started":1}
I20260807 06:22:46.819793 23719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:46.815877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:42955 (local address 127.0.0.1:36130)
0807 06:22:46.816841 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:46.816896 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:46.817335 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:46.818503 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:46.818506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:46.818527 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:46.818946 (+ 419us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:22:46.818949 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:46.818962 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:22:46.818997 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:46.819328 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:46.819342 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:46.819349 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 06:22:46.819474 (+ 125us) client_negotiation.cc:240] Negotiation successful
0807 06:22:46.819559 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"spinlock_wait_cycles":4480,"thread_start_us":146,"threads_started":1}
I20260807 06:22:46.819792 23722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:46.815892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:34049 (local address 127.0.0.1:42424)
0807 06:22:46.816999 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:46.817014 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:46.817132 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:46.817820 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:46.817826 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:46.817902 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:46.818531 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:22:46.818533 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:46.818553 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:22:46.818642 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:46.819328 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:46.819342 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:46.819349 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 06:22:46.819474 (+ 125us) client_negotiation.cc:240] Negotiation successful
0807 06:22:46.819567 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"mutex_wait_us":15,"thread_start_us":145,"threads_started":1}
I20260807 06:22:46.820469 23724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:46.817166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45492 (local address 127.15.254.2:42723)
0807 06:22:46.817507 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:46.817517 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:46.817537 (+ 20us) server_negotiation.cc:407] Connection header received
0807 06:22:46.817584 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:46.817589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:46.817730 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:46.817861 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:46.819606 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:46.819612 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:46.819623 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:22:46.819635 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:46.819678 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:22:46.819714 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:46.819817 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:46.820256 (+ 439us) server_negotiation.cc:299] Negotiation successful
0807 06:22:46.820332 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":112,"threads_started":1}
I20260807 06:22:46.820154 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:46.816733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:42723 (local address 127.0.0.1:45492)
0807 06:22:46.817154 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:46.817166 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:46.817242 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:46.818554 (+ 1312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:46.818557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:46.818572 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:46.819402 (+ 830us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:22:46.819404 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:46.819412 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:22:46.819459 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:46.819868 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:46.819871 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:46.819874 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 06:22:46.819957 (+ 83us) client_negotiation.cc:240] Negotiation successful
0807 06:22:46.820001 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":100,"threads_started":1}
I20260807 06:22:46.821272 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:46.816797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42424 (local address 127.15.254.3:34049)
0807 06:22:46.817296 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:46.817328 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:46.817354 (+ 26us) server_negotiation.cc:407] Connection header received
0807 06:22:46.817411 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:46.817416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:46.817634 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:46.818222 (+ 588us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:46.819102 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:46.819109 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:46.819121 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:22:46.819132 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:46.819173 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:22:46.819202 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:46.819324 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:46.821097 (+ 1773us) server_negotiation.cc:299] Negotiation successful
0807 06:22:46.821157 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":173,"threads_started":1}
I20260807 06:22:46.823282 23234 tablet_service.cc:1467] Processing CreateTablet for tablet 366a991fde494cd5bec4702ddb66629e (DEFAULT_TABLE table=test-table [id=85935b856d714ac08b0bd5dfc67e98b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:46.823302 23415 tablet_service.cc:1467] Processing CreateTablet for tablet 366a991fde494cd5bec4702ddb66629e (DEFAULT_TABLE table=test-table [id=85935b856d714ac08b0bd5dfc67e98b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:46.823606 23616 tablet_service.cc:1467] Processing CreateTablet for tablet 366a991fde494cd5bec4702ddb66629e (DEFAULT_TABLE table=test-table [id=85935b856d714ac08b0bd5dfc67e98b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:46.824520 23415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 366a991fde494cd5bec4702ddb66629e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:46.824513 23234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 366a991fde494cd5bec4702ddb66629e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:46.824760 23616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 366a991fde494cd5bec4702ddb66629e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:46.831286 23727 tablet_bootstrap.cc:492] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435: Bootstrap starting.
I20260807 06:22:46.833016 23729 tablet_bootstrap.cc:492] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5: Bootstrap starting.
I20260807 06:22:46.834227 23727 tablet_bootstrap.cc:654] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:46.834908 23729 tablet_bootstrap.cc:654] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:46.835074 23727 log.cc:824] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:46.835762 23730 tablet_bootstrap.cc:492] T 366a991fde494cd5bec4702ddb66629e P 439e5263aaea4e71928abba3dfa8be0c: Bootstrap starting.
I20260807 06:22:46.835767 23729 log.cc:824] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:46.837424 23727 tablet_bootstrap.cc:492] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435: No bootstrap required, opened a new log
I20260807 06:22:46.837671 23727 ts_tablet_manager.cc:1397] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260807 06:22:46.837874 23730 tablet_bootstrap.cc:654] T 366a991fde494cd5bec4702ddb66629e P 439e5263aaea4e71928abba3dfa8be0c: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:46.838166 23729 tablet_bootstrap.cc:492] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5: No bootstrap required, opened a new log
I20260807 06:22:46.838356 23729 ts_tablet_manager.cc:1397] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260807 06:22:46.838974 23730 log.cc:824] T 366a991fde494cd5bec4702ddb66629e P 439e5263aaea4e71928abba3dfa8be0c: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:46.840734 23730 tablet_bootstrap.cc:492] T 366a991fde494cd5bec4702ddb66629e P 439e5263aaea4e71928abba3dfa8be0c: No bootstrap required, opened a new log
I20260807 06:22:46.840935 23730 ts_tablet_manager.cc:1397] T 366a991fde494cd5bec4702ddb66629e P 439e5263aaea4e71928abba3dfa8be0c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260807 06:22:46.842535 23727 raft_consensus.cc:348] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a15a8509e47afb008de2720b672c5" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42955 } } peers { permanent_uuid: "0933341a5d7641cf9ed58d718739e435" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34049 } } peers { permanent_uuid: "439e5263aaea4e71928abba3dfa8be0c" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 42723 } }
I20260807 06:22:46.842820 23727 raft_consensus.cc:374] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:46.842890 23727 raft_consensus.cc:729] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0933341a5d7641cf9ed58d718739e435, State: Initialized, Role: FOLLOWER
I20260807 06:22:46.842849 23729 raft_consensus.cc:348] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a15a8509e47afb008de2720b672c5" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42955 } } peers { permanent_uuid: "0933341a5d7641cf9ed58d718739e435" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34049 } } peers { permanent_uuid: "439e5263aaea4e71928abba3dfa8be0c" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 42723 } }
I20260807 06:22:46.843102 23729 raft_consensus.cc:374] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:46.843168 23729 raft_consensus.cc:729] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895a15a8509e47afb008de2720b672c5, State: Initialized, Role: FOLLOWER
I20260807 06:22:46.843408 23727 consensus_queue.cc:260] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a15a8509e47afb008de2720b672c5" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42955 } } peers { permanent_uuid: "0933341a5d7641cf9ed58d718739e435" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34049 } } peers { permanent_uuid: "439e5263aaea4e71928abba3dfa8be0c" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 42723 } }
I20260807 06:22:46.843529 23729 consensus_queue.cc:260] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a15a8509e47afb008de2720b672c5" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42955 } } peers { permanent_uuid: "0933341a5d7641cf9ed58d718739e435" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34049 } } peers { permanent_uuid: "439e5263aaea4e71928abba3dfa8be0c" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 42723 } }
I20260807 06:22:46.844159 23701 heartbeater.cc:503] Master 127.15.254.62:37435 was elected leader, sending a full tablet report...
I20260807 06:22:46.844515 23729 ts_tablet_manager.cc:1428] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260807 06:22:46.844906 23730 raft_consensus.cc:348] T 366a991fde494cd5bec4702ddb66629e P 439e5263aaea4e71928abba3dfa8be0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a15a8509e47afb008de2720b672c5" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42955 } } peers { permanent_uuid: "0933341a5d7641cf9ed58d718739e435" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34049 } } peers { permanent_uuid: "439e5263aaea4e71928abba3dfa8be0c" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 42723 } }
I20260807 06:22:46.845167 23730 raft_consensus.cc:374] T 366a991fde494cd5bec4702ddb66629e P 439e5263aaea4e71928abba3dfa8be0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:46.845223 23730 raft_consensus.cc:729] T 366a991fde494cd5bec4702ddb66629e P 439e5263aaea4e71928abba3dfa8be0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 439e5263aaea4e71928abba3dfa8be0c, State: Initialized, Role: FOLLOWER
I20260807 06:22:46.845321 23319 heartbeater.cc:503] Master 127.15.254.62:37435 was elected leader, sending a full tablet report...
I20260807 06:22:46.845695 23730 consensus_queue.cc:260] T 366a991fde494cd5bec4702ddb66629e P 439e5263aaea4e71928abba3dfa8be0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a15a8509e47afb008de2720b672c5" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42955 } } peers { permanent_uuid: "0933341a5d7641cf9ed58d718739e435" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34049 } } peers { permanent_uuid: "439e5263aaea4e71928abba3dfa8be0c" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 42723 } }
I20260807 06:22:46.846386 23506 heartbeater.cc:503] Master 127.15.254.62:37435 was elected leader, sending a full tablet report...
I20260807 06:22:46.846704 23730 ts_tablet_manager.cc:1428] T 366a991fde494cd5bec4702ddb66629e P 439e5263aaea4e71928abba3dfa8be0c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260807 06:22:46.847235 23727 ts_tablet_manager.cc:1428] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435: Time spent starting tablet: real 0.009s user 0.002s sys 0.006s
W20260807 06:22:46.907087 23320 tablet.cc:2369] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:22:47.003225 23736 raft_consensus.cc:482] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:22:47.003533 23736 raft_consensus.cc:504] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a15a8509e47afb008de2720b672c5" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42955 } } peers { permanent_uuid: "0933341a5d7641cf9ed58d718739e435" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34049 } } peers { permanent_uuid: "439e5263aaea4e71928abba3dfa8be0c" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 42723 } }
I20260807 06:22:47.004724 23737 raft_consensus.cc:482] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:22:47.004940 23737 raft_consensus.cc:504] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a15a8509e47afb008de2720b672c5" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42955 } } peers { permanent_uuid: "0933341a5d7641cf9ed58d718739e435" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34049 } } peers { permanent_uuid: "439e5263aaea4e71928abba3dfa8be0c" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 42723 } }
I20260807 06:22:47.005154 23736 leader_election.cc:288] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 895a15a8509e47afb008de2720b672c5 (127.15.254.1:42955), 439e5263aaea4e71928abba3dfa8be0c (127.15.254.2:42723)
I20260807 06:22:47.006206 23737 leader_election.cc:288] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0933341a5d7641cf9ed58d718739e435 (127.15.254.3:34049), 439e5263aaea4e71928abba3dfa8be0c (127.15.254.2:42723)
I20260807 06:22:47.007573 23705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:47.005446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:42955 (local address 127.15.254.3:48181)
0807 06:22:47.005604 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:47.005622 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:47.005719 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:47.006097 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:47.006099 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:47.006115 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:47.006940 (+ 825us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:22:47.006943 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:47.006955 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:22:47.006986 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:47.007298 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:47.007303 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:47.007307 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 06:22:47.007383 (+ 76us) client_negotiation.cc:240] Negotiation successful
0807 06:22:47.007419 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260807 06:22:47.007927 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:47.005537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:48181 (local address 127.15.254.1:42955)
0807 06:22:47.005671 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:47.005675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:47.005693 (+ 18us) server_negotiation.cc:407] Connection header received
0807 06:22:47.005856 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:47.005862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:47.005911 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:47.005986 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:47.007117 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:47.007124 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:47.007130 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:22:47.007144 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:47.007165 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:22:47.007176 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:47.007253 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:47.007738 (+ 485us) server_negotiation.cc:299] Negotiation successful
0807 06:22:47.007785 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 06:22:47.008563 23259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366a991fde494cd5bec4702ddb66629e" candidate_uuid: "0933341a5d7641cf9ed58d718739e435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895a15a8509e47afb008de2720b672c5" is_pre_election: true
I20260807 06:22:47.008787 23724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:47.005930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:55795 (local address 127.15.254.2:42723)
0807 06:22:47.006156 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:47.006161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:47.006274 (+ 113us) server_negotiation.cc:407] Connection header received
0807 06:22:47.006851 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:47.006855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:47.006893 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:47.006956 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:47.008241 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:47.008244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:47.008247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:22:47.008260 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:47.008284 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:22:47.008294 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:47.008418 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:47.008607 (+ 189us) server_negotiation.cc:299] Negotiation successful
0807 06:22:47.008652 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260807 06:22:47.008788 23742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:47.006547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:42723 (local address 127.15.254.1:36879)
0807 06:22:47.007096 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:47.007109 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:47.007172 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:47.007652 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:47.007656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:47.007674 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:47.008123 (+ 449us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:22:47.008126 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:47.008138 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:22:47.008166 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:47.008556 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:47.008560 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:47.008564 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 06:22:47.008626 (+ 62us) client_negotiation.cc:240] Negotiation successful
0807 06:22:47.008660 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":109,"threads_started":1}
I20260807 06:22:47.008868 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:47.006752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:39139 (local address 127.15.254.3:34049)
0807 06:22:47.006945 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:47.006949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:47.006964 (+ 15us) server_negotiation.cc:407] Connection header received
0807 06:22:47.007102 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:47.007105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:47.007134 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:47.007183 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:47.008348 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:47.008354 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:47.008371 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:22:47.008407 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:47.008427 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:22:47.008439 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:47.008557 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:47.008718 (+ 161us) server_negotiation.cc:299] Negotiation successful
0807 06:22:47.008753 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260807 06:22:47.009160 23740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:47.005733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:42723 (local address 127.15.254.3:55795)
0807 06:22:47.006171 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:47.006187 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:47.006716 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:47.007098 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:47.007101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:47.007115 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:47.008055 (+ 940us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:22:47.008058 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:47.008071 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:22:47.008105 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:47.008461 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:47.008467 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:47.008474 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 06:22:47.008994 (+ 520us) client_negotiation.cc:240] Negotiation successful
0807 06:22:47.009029 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":155,"threads_started":1}
I20260807 06:22:47.009172 23744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:47.006402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:34049 (local address 127.15.254.1:39139)
0807 06:22:47.006897 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:47.006911 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:47.006986 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:47.007315 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:47.007320 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:47.007338 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:47.007902 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:22:47.007904 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:47.007917 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:22:47.007975 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:47.008611 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:47.008616 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:47.008619 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 06:22:47.008997 (+ 378us) client_negotiation.cc:240] Negotiation successful
0807 06:22:47.009032 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":118,"threads_started":1}
I20260807 06:22:47.009326 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:47.006685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:36879 (local address 127.15.254.2:42723)
0807 06:22:47.007425 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:47.007434 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:47.007447 (+ 13us) server_negotiation.cc:407] Connection header received
0807 06:22:47.007482 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:47.007485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:47.007517 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:47.007610 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:47.008348 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:47.008351 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:47.008364 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:22:47.008377 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:47.008405 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:22:47.008415 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:47.009108 (+ 693us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:47.009140 (+ 32us) server_negotiation.cc:299] Negotiation successful
0807 06:22:47.009172 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":113,"threads_started":1}
I20260807 06:22:47.008800 23259 raft_consensus.cc:2454] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0933341a5d7641cf9ed58d718739e435 in term 0.
I20260807 06:22:47.009711 23640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366a991fde494cd5bec4702ddb66629e" candidate_uuid: "895a15a8509e47afb008de2720b672c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0933341a5d7641cf9ed58d718739e435" is_pre_election: true
I20260807 06:22:47.009814 23532 leader_election.cc:302] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0933341a5d7641cf9ed58d718739e435, 895a15a8509e47afb008de2720b672c5; no voters:
I20260807 06:22:47.009918 23640 raft_consensus.cc:2454] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 895a15a8509e47afb008de2720b672c5 in term 0.
I20260807 06:22:47.010020 23439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366a991fde494cd5bec4702ddb66629e" candidate_uuid: "0933341a5d7641cf9ed58d718739e435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "439e5263aaea4e71928abba3dfa8be0c" is_pre_election: true
I20260807 06:22:47.010126 23736 raft_consensus.cc:2781] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:22:47.010222 23736 raft_consensus.cc:482] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:22:47.010278 23736 raft_consensus.cc:3037] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:47.010376 23439 raft_consensus.cc:2454] T 366a991fde494cd5bec4702ddb66629e P 439e5263aaea4e71928abba3dfa8be0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0933341a5d7641cf9ed58d718739e435 in term 0.
I20260807 06:22:47.010357 23151 leader_election.cc:302] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0933341a5d7641cf9ed58d718739e435, 895a15a8509e47afb008de2720b672c5; no voters:
I20260807 06:22:47.010037 23440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366a991fde494cd5bec4702ddb66629e" candidate_uuid: "895a15a8509e47afb008de2720b672c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "439e5263aaea4e71928abba3dfa8be0c" is_pre_election: true
I20260807 06:22:47.010646 23737 raft_consensus.cc:2781] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:22:47.010731 23737 raft_consensus.cc:482] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:22:47.010728 23440 raft_consensus.cc:2454] T 366a991fde494cd5bec4702ddb66629e P 439e5263aaea4e71928abba3dfa8be0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 895a15a8509e47afb008de2720b672c5 in term 0.
I20260807 06:22:47.010794 23737 raft_consensus.cc:3037] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:47.011654 23736 raft_consensus.cc:504] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a15a8509e47afb008de2720b672c5" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42955 } } peers { permanent_uuid: "0933341a5d7641cf9ed58d718739e435" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34049 } } peers { permanent_uuid: "439e5263aaea4e71928abba3dfa8be0c" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 42723 } }
I20260807 06:22:47.011996 23737 raft_consensus.cc:504] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a15a8509e47afb008de2720b672c5" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42955 } } peers { permanent_uuid: "0933341a5d7641cf9ed58d718739e435" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34049 } } peers { permanent_uuid: "439e5263aaea4e71928abba3dfa8be0c" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 42723 } }
I20260807 06:22:47.012043 23736 leader_election.cc:288] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435 [CANDIDATE]: Term 1 election: Requested vote from peers 895a15a8509e47afb008de2720b672c5 (127.15.254.1:42955), 439e5263aaea4e71928abba3dfa8be0c (127.15.254.2:42723)
I20260807 06:22:47.012373 23259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366a991fde494cd5bec4702ddb66629e" candidate_uuid: "0933341a5d7641cf9ed58d718739e435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895a15a8509e47afb008de2720b672c5"
I20260807 06:22:47.012401 23440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366a991fde494cd5bec4702ddb66629e" candidate_uuid: "0933341a5d7641cf9ed58d718739e435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "439e5263aaea4e71928abba3dfa8be0c"
I20260807 06:22:47.012531 23737 leader_election.cc:288] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5 [CANDIDATE]: Term 1 election: Requested vote from peers 0933341a5d7641cf9ed58d718739e435 (127.15.254.3:34049), 439e5263aaea4e71928abba3dfa8be0c (127.15.254.2:42723)
I20260807 06:22:47.012542 23440 raft_consensus.cc:3037] T 366a991fde494cd5bec4702ddb66629e P 439e5263aaea4e71928abba3dfa8be0c [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:47.012616 23259 raft_consensus.cc:2379] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0933341a5d7641cf9ed58d718739e435 in current term 1: Already voted for candidate 895a15a8509e47afb008de2720b672c5 in this term.
I20260807 06:22:47.012856 23640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366a991fde494cd5bec4702ddb66629e" candidate_uuid: "895a15a8509e47afb008de2720b672c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0933341a5d7641cf9ed58d718739e435"
I20260807 06:22:47.012966 23439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366a991fde494cd5bec4702ddb66629e" candidate_uuid: "895a15a8509e47afb008de2720b672c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "439e5263aaea4e71928abba3dfa8be0c"
I20260807 06:22:47.013034 23640 raft_consensus.cc:2379] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 895a15a8509e47afb008de2720b672c5 in current term 1: Already voted for candidate 0933341a5d7641cf9ed58d718739e435 in this term.
I20260807 06:22:47.013504 23152 leader_election.cc:302] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 895a15a8509e47afb008de2720b672c5; no voters: 0933341a5d7641cf9ed58d718739e435, 439e5263aaea4e71928abba3dfa8be0c
I20260807 06:22:47.013688 23737 raft_consensus.cc:2726] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260807 06:22:47.013913 23440 raft_consensus.cc:2454] T 366a991fde494cd5bec4702ddb66629e P 439e5263aaea4e71928abba3dfa8be0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0933341a5d7641cf9ed58d718739e435 in term 1.
I20260807 06:22:47.014176 23532 leader_election.cc:302] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0933341a5d7641cf9ed58d718739e435, 439e5263aaea4e71928abba3dfa8be0c; no voters: 895a15a8509e47afb008de2720b672c5
I20260807 06:22:47.014381 23736 raft_consensus.cc:2781] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:47.014652 23736 raft_consensus.cc:686] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435 [term 1 LEADER]: Becoming Leader. State: Replica: 0933341a5d7641cf9ed58d718739e435, State: Running, Role: LEADER
I20260807 06:22:47.014986 23736 consensus_queue.cc:237] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a15a8509e47afb008de2720b672c5" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42955 } } peers { permanent_uuid: "0933341a5d7641cf9ed58d718739e435" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34049 } } peers { permanent_uuid: "439e5263aaea4e71928abba3dfa8be0c" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 42723 } }
I20260807 06:22:47.018240 23046 catalog_manager.cc:5495] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0933341a5d7641cf9ed58d718739e435 (127.15.254.3). New cstate: current_term: 1 leader_uuid: "0933341a5d7641cf9ed58d718739e435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a15a8509e47afb008de2720b672c5" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0933341a5d7641cf9ed58d718739e435" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "439e5263aaea4e71928abba3dfa8be0c" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 42723 } health_report { overall_health: UNKNOWN } } }
W20260807 06:22:47.027848 23703 tablet.cc:2369] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:22:47.057638 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:47.055982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42434 (local address 127.15.254.3:34049)
0807 06:22:47.056138 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:47.056143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:47.056161 (+ 18us) server_negotiation.cc:407] Connection header received
0807 06:22:47.056269 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:47.056271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:47.056336 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:47.056445 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:47.057186 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:47.057189 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:47.057192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:22:47.057211 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:47.057232 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:22:47.057241 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:47.057345 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:47.057432 (+ 87us) server_negotiation.cc:299] Negotiation successful
0807 06:22:47.057478 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 06:22:47.063987 23440 raft_consensus.cc:1261] T 366a991fde494cd5bec4702ddb66629e P 439e5263aaea4e71928abba3dfa8be0c [term 1 FOLLOWER]: Refusing update from remote peer 0933341a5d7641cf9ed58d718739e435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:22:47.063987 23259 raft_consensus.cc:1261] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5 [term 1 FOLLOWER]: Refusing update from remote peer 0933341a5d7641cf9ed58d718739e435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:22:47.064719 23736 consensus_queue.cc:1035] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "439e5263aaea4e71928abba3dfa8be0c" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:22:47.065019 23750 consensus_queue.cc:1035] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "895a15a8509e47afb008de2720b672c5" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:22:47.074699 23753 mvcc.cc:205] Tried to move back new op lower bound from 7315799109885562880 to 7315799109696061440. Current Snapshot: MvccSnapshot[applied={T|T < 7315799109885562880}]
I20260807 06:22:47.085753 23047 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:22:47.085915 23047 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:22:47.088586 23047 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dcb225ce72c94d9fabc68f31ded5081b: Sending DeleteTablet for 3 replicas of tablet 366a991fde494cd5bec4702ddb66629e
I20260807 06:22:47.089365 23616 tablet_service.cc:1514] Processing DeleteTablet for tablet 366a991fde494cd5bec4702ddb66629e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:47 UTC) from {username='slave'} at 127.0.0.1:42424
I20260807 06:22:47.089447 23234 tablet_service.cc:1514] Processing DeleteTablet for tablet 366a991fde494cd5bec4702ddb66629e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:47 UTC) from {username='slave'} at 127.0.0.1:36130
I20260807 06:22:47.089891 23765 tablet_replica.cc:330] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435: stopping tablet replica
I20260807 06:22:47.089995 23767 tablet_replica.cc:330] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5: stopping tablet replica
I20260807 06:22:47.090248 23767 raft_consensus.cc:2229] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:22:47.090248 23765 raft_consensus.cc:2229] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:22:47.090546 23767 raft_consensus.cc:2258] T 366a991fde494cd5bec4702ddb66629e P 895a15a8509e47afb008de2720b672c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:47.090590 23765 raft_consensus.cc:2258] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:47.090564 23415 tablet_service.cc:1514] Processing DeleteTablet for tablet 366a991fde494cd5bec4702ddb66629e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:47 UTC) from {username='slave'} at 127.0.0.1:45492
I20260807 06:22:47.091076 23768 tablet_replica.cc:330] T 366a991fde494cd5bec4702ddb66629e P 439e5263aaea4e71928abba3dfa8be0c: stopping tablet replica
I20260807 06:22:47.091259 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 23131
I20260807 06:22:47.091369 23768 raft_consensus.cc:2229] T 366a991fde494cd5bec4702ddb66629e P 439e5263aaea4e71928abba3dfa8be0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:22:47.091667 23768 raft_consensus.cc:2258] T 366a991fde494cd5bec4702ddb66629e P 439e5263aaea4e71928abba3dfa8be0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:47.093183 23768 ts_tablet_manager.cc:1905] T 366a991fde494cd5bec4702ddb66629e P 439e5263aaea4e71928abba3dfa8be0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:22:47.093184 23765 ts_tablet_manager.cc:1905] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:22:47.096043 23765 ts_tablet_manager.cc:1918] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:22:47.096043 23768 ts_tablet_manager.cc:1918] T 366a991fde494cd5bec4702ddb66629e P 439e5263aaea4e71928abba3dfa8be0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:22:47.096148 23768 log.cc:1196] T 366a991fde494cd5bec4702ddb66629e P 439e5263aaea4e71928abba3dfa8be0c: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/366a991fde494cd5bec4702ddb66629e
I20260807 06:22:47.096148 23765 log.cc:1196] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/366a991fde494cd5bec4702ddb66629e
I20260807 06:22:47.096517 23765 ts_tablet_manager.cc:1939] T 366a991fde494cd5bec4702ddb66629e P 0933341a5d7641cf9ed58d718739e435: Deleting consensus metadata
I20260807 06:22:47.096599 23768 ts_tablet_manager.cc:1939] T 366a991fde494cd5bec4702ddb66629e P 439e5263aaea4e71928abba3dfa8be0c: Deleting consensus metadata
I20260807 06:22:47.097796 23024 catalog_manager.cc:4841] TS 439e5263aaea4e71928abba3dfa8be0c (127.15.254.2:42723): tablet 366a991fde494cd5bec4702ddb66629e (table test-table [id=85935b856d714ac08b0bd5dfc67e98b5]) successfully deleted
I20260807 06:22:47.103179 23023 catalog_manager.cc:4841] TS 0933341a5d7641cf9ed58d718739e435 (127.15.254.3:34049): tablet 366a991fde494cd5bec4702ddb66629e (table test-table [id=85935b856d714ac08b0bd5dfc67e98b5]) successfully deleted
W20260807 06:22:47.113265 23024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.254.1:42955 (error 108)
I20260807 06:22:47.113493 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 23323
I20260807 06:22:47.114715 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:47.114301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:42955 (local address 127.0.0.1:36134)
0807 06:22:47.114477 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:47.114588 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.1:42955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260807 06:22:47.114959 23024 catalog_manager.cc:4584] TS 895a15a8509e47afb008de2720b672c5 (127.15.254.1:42955): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 366a991fde494cd5bec4702ddb66629e: Network error: Client connection negotiation failed: client connection to 127.15.254.1:42955: connect: Connection refused (error 111)
I20260807 06:22:47.128320 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 23510
I20260807 06:22:47.135942 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 23001
2026-08-07T06:22:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1194 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260807 06:22:47.160991 16376 mini_oidc.cc:152] Starting JWKS server
I20260807 06:22:47.162858 16376 webserver.cc:460] Webserver started at https://127.0.0.1:46175/ using document root <none> and password file <none>
I20260807 06:22:47.163045 16376 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 06:22:47.163339 16376 webserver.cc:460] Webserver started at http://127.0.0.1:42371/ using document root <none> and password file <none>
2026-08-07T06:22:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:22:47Z Disabled control of system clock
I20260807 06:22:47.173868 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:46839
--webserver_interface=127.15.254.62
--webserver_port=0
--builtin_ntp_servers=127.15.254.20:36617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:46839
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46175/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786083738959718-16376-0/testchainca.cert with env {}
W20260807 06:22:47.293224 23777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:47.293569 23777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:47.293684 23777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:47.297803 23777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:22:47.297904 23777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:47.297936 23777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:47.297999 23777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:22:47.298041 23777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 06:22:47.298082 23777 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 06:22:47.298120 23777 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46175/jwks.json
I20260807 06:22:47.303613 23777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:36617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:46839
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:46839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46175/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786083738959718-16376-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:47.304771 23777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:47.305846 23777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:47.315310 23790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:47.315479 23788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:47.315299 23786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:47.315729 23777 server_base.cc:1034] running on GCE node
I20260807 06:22:47.316363 23777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:47.317677 23777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:47.318892 23777 hybrid_clock.cc:648] HybridClock initialized: now 1786083767318870 us; error 33 us; skew 500 ppm
I20260807 06:22:47.321322 23777 webserver.cc:460] Webserver started at http://127.15.254.62:46327/ using document root <none> and password file <none>
I20260807 06:22:47.322001 23777 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:47.322113 23777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:47.322378 23777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:47.324810 23777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "14209fe8e81e41c2b5f11a4f817f263c"
format_stamp: "Formatted at 2026-08-07 06:22:47 on dist-test-slave-44d2"
server_key: "db01726ab550be1854b755b4bf8b3c46"
server_key_iv: "04a590a1a10f61c1bdc8bac6858562c5"
server_key_version: "encryptionkey@0"
I20260807 06:22:47.325438 23777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "14209fe8e81e41c2b5f11a4f817f263c"
format_stamp: "Formatted at 2026-08-07 06:22:47 on dist-test-slave-44d2"
server_key: "db01726ab550be1854b755b4bf8b3c46"
server_key_iv: "04a590a1a10f61c1bdc8bac6858562c5"
server_key_version: "encryptionkey@0"
I20260807 06:22:47.329469 23777 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 06:22:47.332247 23799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:47.333952 23777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 06:22:47.334126 23777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "14209fe8e81e41c2b5f11a4f817f263c"
format_stamp: "Formatted at 2026-08-07 06:22:47 on dist-test-slave-44d2"
server_key: "db01726ab550be1854b755b4bf8b3c46"
server_key_iv: "04a590a1a10f61c1bdc8bac6858562c5"
server_key_version: "encryptionkey@0"
I20260807 06:22:47.334280 23777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:47.347738 23777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:47.348580 23777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:47.348814 23777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:47.357331 23777 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:46839
I20260807 06:22:47.357352 23880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:46839 every 8 connection(s)
I20260807 06:22:47.358418 23777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:22:47.360401 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 23777
I20260807 06:22:47.360581 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:22:47.360898 16376 external_mini_cluster.cc:1428] Setting key f12b58409f7a94327e9d7f9e95a1166c
I20260807 06:22:47.362123 23881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:47.368022 23881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14209fe8e81e41c2b5f11a4f817f263c: Bootstrap starting.
I20260807 06:22:47.370517 23881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14209fe8e81e41c2b5f11a4f817f263c: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:47.370965 23885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:47.362396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37212 (local address 127.15.254.62:46839)
0807 06:22:47.363172 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:47.363191 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:47.363233 (+ 42us) server_negotiation.cc:407] Connection header received
0807 06:22:47.363936 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:47.363953 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:47.364360 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:47.364801 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:47.365272 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:47.366334 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:47.367342 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:47.367865 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:47.368053 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:47.369047 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:47.369062 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:47.369076 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:22:47.369478 (+ 402us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:47.369548 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:22:47.369570 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:47.369693 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:47.369853 (+ 160us) server_negotiation.cc:299] Negotiation successful
0807 06:22:47.370133 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":164,"threads_started":1}
I20260807 06:22:47.371430 23881 log.cc:824] T 00000000000000000000000000000000 P 14209fe8e81e41c2b5f11a4f817f263c: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:47.373821 23881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14209fe8e81e41c2b5f11a4f817f263c: No bootstrap required, opened a new log
I20260807 06:22:47.374143 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:22:47.377682 23881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14209fe8e81e41c2b5f11a4f817f263c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14209fe8e81e41c2b5f11a4f817f263c" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 46839 } }
I20260807 06:22:47.377894 23881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14209fe8e81e41c2b5f11a4f817f263c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:47.377975 23881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14209fe8e81e41c2b5f11a4f817f263c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14209fe8e81e41c2b5f11a4f817f263c, State: Initialized, Role: FOLLOWER
I20260807 06:22:47.378437 23881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14209fe8e81e41c2b5f11a4f817f263c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14209fe8e81e41c2b5f11a4f817f263c" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 46839 } }
I20260807 06:22:47.378592 23881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14209fe8e81e41c2b5f11a4f817f263c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:22:47.378661 23881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14209fe8e81e41c2b5f11a4f817f263c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:22:47.378789 23881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14209fe8e81e41c2b5f11a4f817f263c [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:47.379813 23881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14209fe8e81e41c2b5f11a4f817f263c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14209fe8e81e41c2b5f11a4f817f263c" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 46839 } }
I20260807 06:22:47.380154 23881 leader_election.cc:302] T 00000000000000000000000000000000 P 14209fe8e81e41c2b5f11a4f817f263c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14209fe8e81e41c2b5f11a4f817f263c; no voters:
I20260807 06:22:47.380460 23881 leader_election.cc:288] T 00000000000000000000000000000000 P 14209fe8e81e41c2b5f11a4f817f263c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:22:47.380636 23890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14209fe8e81e41c2b5f11a4f817f263c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:47.380860 23890 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14209fe8e81e41c2b5f11a4f817f263c [term 1 LEADER]: Becoming Leader. State: Replica: 14209fe8e81e41c2b5f11a4f817f263c, State: Running, Role: LEADER
I20260807 06:22:47.381206 23890 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14209fe8e81e41c2b5f11a4f817f263c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14209fe8e81e41c2b5f11a4f817f263c" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 46839 } }
I20260807 06:22:47.381764 23881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14209fe8e81e41c2b5f11a4f817f263c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:22:47.382910 23894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14209fe8e81e41c2b5f11a4f817f263c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14209fe8e81e41c2b5f11a4f817f263c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14209fe8e81e41c2b5f11a4f817f263c" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 46839 } } }
I20260807 06:22:47.383080 23894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14209fe8e81e41c2b5f11a4f817f263c [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:47.383411 23895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14209fe8e81e41c2b5f11a4f817f263c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14209fe8e81e41c2b5f11a4f817f263c. Latest consensus state: current_term: 1 leader_uuid: "14209fe8e81e41c2b5f11a4f817f263c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14209fe8e81e41c2b5f11a4f817f263c" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 46839 } } }
I20260807 06:22:47.383486 23909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:22:47.386619 23895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14209fe8e81e41c2b5f11a4f817f263c [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:47.386976 23909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:22:47.393700 23909 catalog_manager.cc:1347] Generated new cluster ID: e473a55a830748abbc0f80e3fab431ed
I20260807 06:22:47.393797 23909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:22:47.435469 23909 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:22:47.436532 23909 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:22:47.444684 23909 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14209fe8e81e41c2b5f11a4f817f263c: Generated new TSK 0
I20260807 06:22:47.445443 23909 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260807 06:22:47.479454 23929 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260807 06:22:47.479676 23929 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 06:22:47.480257 23885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:47.462208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37228 (local address 127.15.254.62:46839)
0807 06:22:47.462396 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:47.462401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:47.462619 (+ 218us) server_negotiation.cc:407] Connection header received
0807 06:22:47.462865 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:47.462869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:47.462927 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:47.463021 (+ 94us) server_negotiation.cc:226] Negotiated authn=JWT
0807 06:22:47.463373 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:47.463957 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:47.465273 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:47.465508 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:47.465612 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:47.465996 (+ 384us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 06:22:47.479820 (+ 13824us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46175/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0807 06:22:47.480132 (+ 312us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37228: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46175/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":61}
W20260807 06:22:47.480381 23885 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37228: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46175/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260807 06:22:47.482218 23928 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:22:47.462105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:46839 (local address 127.0.0.1:37228)
0807 06:22:47.462514 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:47.462530 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:47.462671 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:47.463066 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:47.463069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:47.463078 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0807 06:22:47.463231 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:47.463239 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:47.464110 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:47.464115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:47.465149 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:47.465160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:47.465634 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:47.465638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:47.465739 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:47.465795 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 06:22:47.480122 (+ 14327us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46175/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0807 06:22:47.480405 (+ 283us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.254.62:46839: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46175/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":117,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.15.254.62:46839: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46175/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260807 06:22:47.483312 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 23777
2026-08-07T06:22:47Z chronyd exiting
I20260807 06:22:47.529780 16376 test_util.cc:175] -----------------------------------------------
I20260807 06:22:47.529907 16376 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786083738959718-16376-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (370 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260807 06:22:47.531098 16376 mini_oidc.cc:152] Starting JWKS server
I20260807 06:22:47.532011 16376 webserver.cc:460] Webserver started at https://127.0.0.1:38709/ using document root <none> and password file <none>
I20260807 06:22:47.532145 16376 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 06:22:47.532449 16376 webserver.cc:460] Webserver started at http://127.0.0.1:41101/ using document root <none> and password file <none>
2026-08-07T06:22:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:22:47Z Disabled control of system clock
I20260807 06:22:47.542893 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:45741
--webserver_interface=127.15.254.62
--webserver_port=0
--builtin_ntp_servers=127.15.254.20:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:45741
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38709/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786083738959718-16376-0/testchainca.cert with env {}
W20260807 06:22:47.658141 23936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:47.658449 23936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:47.658519 23936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:47.662475 23936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:22:47.662565 23936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:47.662595 23936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:47.662637 23936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:22:47.662667 23936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 06:22:47.662704 23936 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 06:22:47.662736 23936 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38709/jwks.json
I20260807 06:22:47.667915 23936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:45741
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:45741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38709/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786083738959718-16376-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:47.669051 23936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:47.670174 23936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:47.679395 23945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:47.679427 23944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:47.679483 23936 server_base.cc:1034] running on GCE node
W20260807 06:22:47.679395 23947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:47.680186 23936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:47.681421 23936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:47.682663 23936 hybrid_clock.cc:648] HybridClock initialized: now 1786083767682644 us; error 27 us; skew 500 ppm
I20260807 06:22:47.684722 23936 webserver.cc:460] Webserver started at http://127.15.254.62:36879/ using document root <none> and password file <none>
I20260807 06:22:47.685321 23936 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:47.685420 23936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:47.685664 23936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:47.687745 23936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "8e9593930cc04ac5980ff60942af4f3d"
format_stamp: "Formatted at 2026-08-07 06:22:47 on dist-test-slave-44d2"
server_key: "871cb20674f7cbd4158235e46d04e1e5"
server_key_iv: "a73cfd4cc3d350dd6a40a47201672fed"
server_key_version: "encryptionkey@0"
I20260807 06:22:47.688324 23936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "8e9593930cc04ac5980ff60942af4f3d"
format_stamp: "Formatted at 2026-08-07 06:22:47 on dist-test-slave-44d2"
server_key: "871cb20674f7cbd4158235e46d04e1e5"
server_key_iv: "a73cfd4cc3d350dd6a40a47201672fed"
server_key_version: "encryptionkey@0"
I20260807 06:22:47.692184 23936 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260807 06:22:47.694875 23955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:47.696292 23936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:22:47.696470 23936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "8e9593930cc04ac5980ff60942af4f3d"
format_stamp: "Formatted at 2026-08-07 06:22:47 on dist-test-slave-44d2"
server_key: "871cb20674f7cbd4158235e46d04e1e5"
server_key_iv: "a73cfd4cc3d350dd6a40a47201672fed"
server_key_version: "encryptionkey@0"
I20260807 06:22:47.696655 23936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:47.713850 23936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:47.714653 23936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:47.714879 23936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:47.722627 23936 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:45741
I20260807 06:22:47.722657 24032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:45741 every 8 connection(s)
I20260807 06:22:47.723713 23936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:22:47.727212 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:47.729352 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 23936
I20260807 06:22:47.729489 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:22:47.729794 16376 external_mini_cluster.cc:1428] Setting key ad36982c5edde1fe3fa81fce472ecbcf
I20260807 06:22:47.733649 24034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e9593930cc04ac5980ff60942af4f3d: Bootstrap starting.
I20260807 06:22:47.735924 24034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e9593930cc04ac5980ff60942af4f3d: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:47.736758 24034 log.cc:824] T 00000000000000000000000000000000 P 8e9593930cc04ac5980ff60942af4f3d: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:47.739235 24034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e9593930cc04ac5980ff60942af4f3d: No bootstrap required, opened a new log
I20260807 06:22:47.739894 24041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:47.731200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40856 (local address 127.15.254.62:45741)
0807 06:22:47.731917 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:47.731934 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:47.731972 (+ 38us) server_negotiation.cc:407] Connection header received
0807 06:22:47.732752 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:47.732772 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:47.733239 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:47.733583 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:47.734032 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:47.734986 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:47.735921 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:47.736461 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:47.736696 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:47.737696 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:47.737703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:47.737720 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:22:47.738219 (+ 499us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:47.738290 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:22:47.738318 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:47.738428 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:47.738588 (+ 160us) server_negotiation.cc:299] Negotiation successful
0807 06:22:47.738892 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":129,"threads_started":1}
I20260807 06:22:47.742816 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:22:47.743116 24034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e9593930cc04ac5980ff60942af4f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9593930cc04ac5980ff60942af4f3d" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 45741 } }
I20260807 06:22:47.743397 24034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e9593930cc04ac5980ff60942af4f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:47.743456 24034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e9593930cc04ac5980ff60942af4f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e9593930cc04ac5980ff60942af4f3d, State: Initialized, Role: FOLLOWER
I20260807 06:22:47.743906 24034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e9593930cc04ac5980ff60942af4f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9593930cc04ac5980ff60942af4f3d" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 45741 } }
I20260807 06:22:47.744076 24034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e9593930cc04ac5980ff60942af4f3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:22:47.744150 24034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e9593930cc04ac5980ff60942af4f3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:22:47.744251 24034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e9593930cc04ac5980ff60942af4f3d [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:47.745208 24034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e9593930cc04ac5980ff60942af4f3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9593930cc04ac5980ff60942af4f3d" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 45741 } }
I20260807 06:22:47.745510 24034 leader_election.cc:302] T 00000000000000000000000000000000 P 8e9593930cc04ac5980ff60942af4f3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e9593930cc04ac5980ff60942af4f3d; no voters:
I20260807 06:22:47.745769 24034 leader_election.cc:288] T 00000000000000000000000000000000 P 8e9593930cc04ac5980ff60942af4f3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:22:47.745900 24044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e9593930cc04ac5980ff60942af4f3d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:47.746214 24044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e9593930cc04ac5980ff60942af4f3d [term 1 LEADER]: Becoming Leader. State: Replica: 8e9593930cc04ac5980ff60942af4f3d, State: Running, Role: LEADER
I20260807 06:22:47.746523 24044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e9593930cc04ac5980ff60942af4f3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9593930cc04ac5980ff60942af4f3d" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 45741 } }
I20260807 06:22:47.747030 24034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e9593930cc04ac5980ff60942af4f3d [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:22:47.748291 24046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e9593930cc04ac5980ff60942af4f3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e9593930cc04ac5980ff60942af4f3d. Latest consensus state: current_term: 1 leader_uuid: "8e9593930cc04ac5980ff60942af4f3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9593930cc04ac5980ff60942af4f3d" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 45741 } } }
I20260807 06:22:47.748332 24045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e9593930cc04ac5980ff60942af4f3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e9593930cc04ac5980ff60942af4f3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9593930cc04ac5980ff60942af4f3d" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 45741 } } }
I20260807 06:22:47.748435 24045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e9593930cc04ac5980ff60942af4f3d [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:47.748435 24046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e9593930cc04ac5980ff60942af4f3d [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:47.748945 24057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:22:47.752394 24057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:22:47.758119 24057 catalog_manager.cc:1347] Generated new cluster ID: 79caf0ec9a254f4d8f100ae0aa978912
I20260807 06:22:47.758193 24057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:22:47.764050 24057 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:22:47.765226 24057 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:22:47.774027 24057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e9593930cc04ac5980ff60942af4f3d: Generated new TSK 0
I20260807 06:22:47.774729 24057 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260807 06:22:47.817444 24078 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260807 06:22:47.817646 24078 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 06:22:47.818323 24041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:47.791847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40864 (local address 127.15.254.62:45741)
0807 06:22:47.792037 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:47.792044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:47.792254 (+ 210us) server_negotiation.cc:407] Connection header received
0807 06:22:47.792430 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:47.792432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:47.792501 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:47.792664 (+ 163us) server_negotiation.cc:226] Negotiated authn=JWT
0807 06:22:47.792946 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:47.793503 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:47.794770 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:47.795000 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:47.795089 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:47.795431 (+ 342us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 06:22:47.817853 (+ 22422us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38709/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0807 06:22:47.818169 (+ 316us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40864: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38709/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":71}
W20260807 06:22:47.818410 24041 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40864: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38709/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260807 06:22:47.818670 24077 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:22:47.791745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:45741 (local address 127.0.0.1:40864)
0807 06:22:47.792125 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:47.792141 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:47.792283 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:47.792683 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:47.792687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:47.792696 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0807 06:22:47.792823 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:47.792830 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:47.793629 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:47.793633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:47.794645 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:47.794656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:47.795119 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:47.795123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:47.795215 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:47.795246 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 06:22:47.818106 (+ 22860us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38709/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0807 06:22:47.818347 (+ 241us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.254.62:45741: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38709/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":151,"threads_started":1}
I20260807 06:22:47.820628 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 23936
2026-08-07T06:22:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (339 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260807 06:22:47.871129 16376 mini_oidc.cc:152] Starting JWKS server
I20260807 06:22:47.872555 16376 webserver.cc:460] Webserver started at https://127.0.0.1:37875/ using document root <none> and password file <none>
I20260807 06:22:47.872684 16376 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 06:22:47.872974 16376 webserver.cc:460] Webserver started at http://127.0.0.1:36815/ using document root <none> and password file <none>
2026-08-07T06:22:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:22:47Z Disabled control of system clock
I20260807 06:22:47.884907 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:41311
--webserver_interface=127.15.254.62
--webserver_port=0
--builtin_ntp_servers=127.15.254.20:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:41311
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37875/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260807 06:22:48.002449 24086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:48.002764 24086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:48.002805 24086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:48.006942 24086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:22:48.007017 24086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:48.007033 24086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:48.007052 24086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:22:48.007068 24086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 06:22:48.007084 24086 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 06:22:48.007098 24086 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37875/jwks.json
I20260807 06:22:48.011937 24086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:41311
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:41311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37875/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:48.013195 24086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:48.014616 24086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:48.023543 24092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:48.023556 24095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:48.023603 24093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:48.023615 24086 server_base.cc:1034] running on GCE node
I20260807 06:22:48.024317 24086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:48.025478 24086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:48.026643 24086 hybrid_clock.cc:648] HybridClock initialized: now 1786083768026614 us; error 40 us; skew 500 ppm
I20260807 06:22:48.028739 24086 webserver.cc:460] Webserver started at http://127.15.254.62:39955/ using document root <none> and password file <none>
I20260807 06:22:48.029389 24086 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:48.029465 24086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:48.029714 24086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:48.031944 24086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "065522021264408e8bf546548652ebf1"
format_stamp: "Formatted at 2026-08-07 06:22:48 on dist-test-slave-44d2"
server_key: "6a0462f412159ec204f40f24130c7684"
server_key_iv: "6c39d9de8ceb0a5d16d3a6f0cea49de0"
server_key_version: "encryptionkey@0"
I20260807 06:22:48.032667 24086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "065522021264408e8bf546548652ebf1"
format_stamp: "Formatted at 2026-08-07 06:22:48 on dist-test-slave-44d2"
server_key: "6a0462f412159ec204f40f24130c7684"
server_key_iv: "6c39d9de8ceb0a5d16d3a6f0cea49de0"
server_key_version: "encryptionkey@0"
I20260807 06:22:48.036787 24086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 06:22:48.039479 24107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:48.040940 24086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:22:48.041097 24086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "065522021264408e8bf546548652ebf1"
format_stamp: "Formatted at 2026-08-07 06:22:48 on dist-test-slave-44d2"
server_key: "6a0462f412159ec204f40f24130c7684"
server_key_iv: "6c39d9de8ceb0a5d16d3a6f0cea49de0"
server_key_version: "encryptionkey@0"
I20260807 06:22:48.041196 24086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:48.057832 24086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:48.058535 24086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:48.058736 24086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:48.066231 24086 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:41311
I20260807 06:22:48.066241 24183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:41311 every 8 connection(s)
I20260807 06:22:48.067178 24086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:22:48.070570 24184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:48.071189 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 24086
I20260807 06:22:48.071329 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:22:48.071626 16376 external_mini_cluster.cc:1428] Setting key 402e48de383fb4e82ede250e39265cae
I20260807 06:22:48.076875 24184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 065522021264408e8bf546548652ebf1: Bootstrap starting.
I20260807 06:22:48.079566 24184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 065522021264408e8bf546548652ebf1: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:48.080332 24184 log.cc:824] T 00000000000000000000000000000000 P 065522021264408e8bf546548652ebf1: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:48.081641 24188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:48.073188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46266 (local address 127.15.254.62:41311)
0807 06:22:48.073906 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:48.073930 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:48.073975 (+ 45us) server_negotiation.cc:407] Connection header received
0807 06:22:48.074668 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:48.074685 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:48.075083 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:48.075434 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:48.075860 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:48.076898 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:48.077843 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:48.078313 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:48.078564 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:48.079507 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:48.079512 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:48.079526 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:22:48.079974 (+ 448us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:48.080033 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:22:48.080054 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:48.080180 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:48.080368 (+ 188us) server_negotiation.cc:299] Negotiation successful
0807 06:22:48.080739 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":136,"threads_started":1}
I20260807 06:22:48.082440 24184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 065522021264408e8bf546548652ebf1: No bootstrap required, opened a new log
I20260807 06:22:48.084448 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:22:48.086408 24184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 065522021264408e8bf546548652ebf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065522021264408e8bf546548652ebf1" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41311 } }
I20260807 06:22:48.086627 24184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 065522021264408e8bf546548652ebf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:48.086670 24184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 065522021264408e8bf546548652ebf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 065522021264408e8bf546548652ebf1, State: Initialized, Role: FOLLOWER
I20260807 06:22:48.087018 24184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 065522021264408e8bf546548652ebf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065522021264408e8bf546548652ebf1" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41311 } }
I20260807 06:22:48.087123 24184 raft_consensus.cc:388] T 00000000000000000000000000000000 P 065522021264408e8bf546548652ebf1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:22:48.087168 24184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 065522021264408e8bf546548652ebf1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:22:48.087236 24184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 065522021264408e8bf546548652ebf1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:48.088043 24184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 065522021264408e8bf546548652ebf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065522021264408e8bf546548652ebf1" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41311 } }
I20260807 06:22:48.088326 24184 leader_election.cc:302] T 00000000000000000000000000000000 P 065522021264408e8bf546548652ebf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 065522021264408e8bf546548652ebf1; no voters:
I20260807 06:22:48.088639 24184 leader_election.cc:288] T 00000000000000000000000000000000 P 065522021264408e8bf546548652ebf1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:22:48.088764 24190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 065522021264408e8bf546548652ebf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:48.089023 24190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 065522021264408e8bf546548652ebf1 [term 1 LEADER]: Becoming Leader. State: Replica: 065522021264408e8bf546548652ebf1, State: Running, Role: LEADER
I20260807 06:22:48.089326 24190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 065522021264408e8bf546548652ebf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065522021264408e8bf546548652ebf1" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41311 } }
I20260807 06:22:48.089875 24184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 065522021264408e8bf546548652ebf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:22:48.090996 24191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 065522021264408e8bf546548652ebf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "065522021264408e8bf546548652ebf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065522021264408e8bf546548652ebf1" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41311 } } }
I20260807 06:22:48.091148 24191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 065522021264408e8bf546548652ebf1 [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:48.091425 24192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 065522021264408e8bf546548652ebf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 065522021264408e8bf546548652ebf1. Latest consensus state: current_term: 1 leader_uuid: "065522021264408e8bf546548652ebf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065522021264408e8bf546548652ebf1" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41311 } } }
I20260807 06:22:48.091543 24192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 065522021264408e8bf546548652ebf1 [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:48.091878 24201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:22:48.094508 24201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:22:48.099701 24201 catalog_manager.cc:1347] Generated new cluster ID: cde8d1af446d44f2b83838cc567e28c8
I20260807 06:22:48.099769 24201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:22:48.121572 24201 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:22:48.122613 24201 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:22:48.131994 24201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 065522021264408e8bf546548652ebf1: Generated new TSK 0
I20260807 06:22:48.132937 24201 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:22:48.156270 24188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:48.150543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46282 (local address 127.15.254.62:41311)
0807 06:22:48.150694 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:48.150700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:48.150865 (+ 165us) server_negotiation.cc:407] Connection header received
0807 06:22:48.151023 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:48.151027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:48.151093 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:48.151225 (+ 132us) server_negotiation.cc:226] Negotiated authn=JWT
0807 06:22:48.151488 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:48.152385 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:48.153384 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:48.153760 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:48.153866 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:48.154234 (+ 368us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 06:22:48.155832 (+ 1598us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37875/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0807 06:22:48.156102 (+ 270us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46282: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37875/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":57}
W20260807 06:22:48.156355 24188 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46282: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37875/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260807 06:22:48.157892 24221 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:22:48.150413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41311 (local address 127.0.0.1:46282)
0807 06:22:48.150774 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:48.150788 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:48.150894 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:48.151231 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:48.151233 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:48.151240 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0807 06:22:48.151371 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:48.151376 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:48.152594 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:48.152598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:48.153253 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:48.153262 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:48.153895 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:48.153898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:48.153970 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:48.153996 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 06:22:48.156006 (+ 2010us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37875/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0807 06:22:48.157713 (+ 1707us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.254.62:41311: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37875/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":1482,"thread_start_us":123,"threads_started":1}
I20260807 06:22:48.158680 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 24086
E20260807 06:22:48.171634 24224 webserver.cc:533] Webserver: sslize failed:
E20260807 06:22:48.171707 24224 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-07T06:22:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (332 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-07T06:22:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:22:48Z Disabled control of system clock
I20260807 06:22:48.213224 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:38055
--webserver_interface=127.15.254.62
--webserver_port=0
--builtin_ntp_servers=127.15.254.20:35217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:38055
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786083738959718-16376-0/testchainca.cert with env {}
W20260807 06:22:48.332041 24228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:48.332297 24228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:48.332338 24228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:48.336412 24228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:22:48.336468 24228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:48.336508 24228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:48.336575 24228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:22:48.336603 24228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 06:22:48.336642 24228 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 06:22:48.336666 24228 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260807 06:22:48.341995 24228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:38055
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:38055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786083738959718-16376-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:48.343080 24228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:48.344084 24228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:48.353277 24235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:48.353314 24238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:48.353312 24236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:48.353547 24228 server_base.cc:1034] running on GCE node
I20260807 06:22:48.354423 24228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:48.355611 24228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:48.356775 24228 hybrid_clock.cc:648] HybridClock initialized: now 1786083768356745 us; error 42 us; skew 500 ppm
I20260807 06:22:48.358747 24228 webserver.cc:460] Webserver started at http://127.15.254.62:37775/ using document root <none> and password file <none>
I20260807 06:22:48.359387 24228 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:48.359483 24228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:48.359733 24228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:48.361943 24228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "9da3ef840b2e49b388b089cb841b7e5f"
format_stamp: "Formatted at 2026-08-07 06:22:48 on dist-test-slave-44d2"
server_key: "112c7c1a2e386f0905f104b7901b2a76"
server_key_iv: "57255f7a2c1375de98887ebeee4cd345"
server_key_version: "encryptionkey@0"
I20260807 06:22:48.362553 24228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "9da3ef840b2e49b388b089cb841b7e5f"
format_stamp: "Formatted at 2026-08-07 06:22:48 on dist-test-slave-44d2"
server_key: "112c7c1a2e386f0905f104b7901b2a76"
server_key_iv: "57255f7a2c1375de98887ebeee4cd345"
server_key_version: "encryptionkey@0"
I20260807 06:22:48.366278 24228 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 06:22:48.368815 24246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:48.370043 24228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 06:22:48.370199 24228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "9da3ef840b2e49b388b089cb841b7e5f"
format_stamp: "Formatted at 2026-08-07 06:22:48 on dist-test-slave-44d2"
server_key: "112c7c1a2e386f0905f104b7901b2a76"
server_key_iv: "57255f7a2c1375de98887ebeee4cd345"
server_key_version: "encryptionkey@0"
I20260807 06:22:48.370317 24228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:48.389206 24228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:48.389988 24228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:48.390159 24228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:48.397266 24228 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:38055
I20260807 06:22:48.397296 24332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:38055 every 8 connection(s)
I20260807 06:22:48.398298 24228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:22:48.399693 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 24228
I20260807 06:22:48.399834 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:22:48.400158 16376 external_mini_cluster.cc:1428] Setting key 3b065630041245232fdb2e9dba31005c
I20260807 06:22:48.401818 24333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:48.408852 24333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9da3ef840b2e49b388b089cb841b7e5f: Bootstrap starting.
I20260807 06:22:48.409636 24338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:48.401542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51002 (local address 127.15.254.62:38055)
0807 06:22:48.402176 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:48.402193 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:48.402230 (+ 37us) server_negotiation.cc:407] Connection header received
0807 06:22:48.402873 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:48.402890 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:48.403332 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:48.403624 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:48.403957 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:48.405014 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:48.406199 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:48.406669 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:48.406789 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:48.407736 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:48.407740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:48.407754 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:22:48.408165 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:48.408224 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:22:48.408244 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:48.408322 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:48.408562 (+ 240us) server_negotiation.cc:299] Negotiation successful
0807 06:22:48.408863 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":148,"threads_started":1}
I20260807 06:22:48.411522 24333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9da3ef840b2e49b388b089cb841b7e5f: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:48.412420 24333 log.cc:824] T 00000000000000000000000000000000 P 9da3ef840b2e49b388b089cb841b7e5f: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:48.412683 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:22:48.414763 24333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9da3ef840b2e49b388b089cb841b7e5f: No bootstrap required, opened a new log
I20260807 06:22:48.418576 24333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9da3ef840b2e49b388b089cb841b7e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da3ef840b2e49b388b089cb841b7e5f" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 38055 } }
I20260807 06:22:48.418886 24333 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9da3ef840b2e49b388b089cb841b7e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:48.418972 24333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9da3ef840b2e49b388b089cb841b7e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da3ef840b2e49b388b089cb841b7e5f, State: Initialized, Role: FOLLOWER
I20260807 06:22:48.419474 24333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9da3ef840b2e49b388b089cb841b7e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da3ef840b2e49b388b089cb841b7e5f" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 38055 } }
I20260807 06:22:48.419641 24333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9da3ef840b2e49b388b089cb841b7e5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:22:48.419729 24333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9da3ef840b2e49b388b089cb841b7e5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:22:48.419842 24333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9da3ef840b2e49b388b089cb841b7e5f [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:48.421221 24333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9da3ef840b2e49b388b089cb841b7e5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da3ef840b2e49b388b089cb841b7e5f" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 38055 } }
I20260807 06:22:48.421545 24333 leader_election.cc:302] T 00000000000000000000000000000000 P 9da3ef840b2e49b388b089cb841b7e5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9da3ef840b2e49b388b089cb841b7e5f; no voters:
I20260807 06:22:48.421839 24333 leader_election.cc:288] T 00000000000000000000000000000000 P 9da3ef840b2e49b388b089cb841b7e5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:22:48.421978 24340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9da3ef840b2e49b388b089cb841b7e5f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:48.422263 24340 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9da3ef840b2e49b388b089cb841b7e5f [term 1 LEADER]: Becoming Leader. State: Replica: 9da3ef840b2e49b388b089cb841b7e5f, State: Running, Role: LEADER
I20260807 06:22:48.422609 24340 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9da3ef840b2e49b388b089cb841b7e5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da3ef840b2e49b388b089cb841b7e5f" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 38055 } }
I20260807 06:22:48.423177 24333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9da3ef840b2e49b388b089cb841b7e5f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:22:48.424398 24342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9da3ef840b2e49b388b089cb841b7e5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9da3ef840b2e49b388b089cb841b7e5f. Latest consensus state: current_term: 1 leader_uuid: "9da3ef840b2e49b388b089cb841b7e5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da3ef840b2e49b388b089cb841b7e5f" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 38055 } } }
I20260807 06:22:48.424584 24342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9da3ef840b2e49b388b089cb841b7e5f [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:48.424862 24341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9da3ef840b2e49b388b089cb841b7e5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9da3ef840b2e49b388b089cb841b7e5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da3ef840b2e49b388b089cb841b7e5f" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 38055 } } }
I20260807 06:22:48.424966 24341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9da3ef840b2e49b388b089cb841b7e5f [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:48.425086 24350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:22:48.428459 24350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:22:48.435395 24350 catalog_manager.cc:1347] Generated new cluster ID: 9b202ee9da014ff68759577a5d7fc238
I20260807 06:22:48.435487 24350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:22:48.446518 24350 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:22:48.447428 24350 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:22:48.453276 24350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9da3ef840b2e49b388b089cb841b7e5f: Generated new TSK 0
I20260807 06:22:48.454058 24350 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:22:48.567962 24338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:48.470193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51010 (local address 127.15.254.62:38055)
0807 06:22:48.470364 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:48.470369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:48.470616 (+ 247us) server_negotiation.cc:407] Connection header received
0807 06:22:48.470782 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:48.470785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:48.470841 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:48.470948 (+ 107us) server_negotiation.cc:226] Negotiated authn=JWT
0807 06:22:48.471244 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:48.471794 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:48.473136 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:48.473405 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:48.473526 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:48.473918 (+ 392us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 06:22:48.567486 (+ 93568us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0807 06:22:48.567817 (+ 331us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51010: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":64}
W20260807 06:22:48.568076 24338 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51010: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260807 06:22:48.568226 24371 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:22:48.470121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:38055 (local address 127.0.0.1:51010)
0807 06:22:48.470479 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:48.470495 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:48.470632 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:48.470973 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:48.470977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:48.470985 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0807 06:22:48.471123 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:48.471129 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:48.471917 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:48.471921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:48.473003 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:48.473014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:48.473555 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:48.473559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:48.473665 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:48.473699 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 06:22:48.567808 (+ 94109us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0807 06:22:48.568035 (+ 227us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.254.62:38055: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":147,"threads_started":1}
I20260807 06:22:48.568976 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 24228
2026-08-07T06:22:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (393 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (134 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (131 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:22:48 dist-test-slave-44d2 krb5kdc[24388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:22:48 dist-test-slave-44d2 krb5kdc[24388](info): set up 2 sockets
Aug 07 06:22:48 dist-test-slave-44d2 krb5kdc[24388](info): commencing operation
krb5kdc: starting...
W20260807 06:22:50.077772 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.215s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:22:50 dist-test-slave-44d2 krb5kdc[24388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:22:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:22:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:22:50.209829 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:34415
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:42261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:34415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:50.328066 24405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:50.328348 24405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:50.328389 24405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:50.332451 24405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:22:50.332535 24405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:50.332556 24405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:50.332597 24405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:22:50.332661 24405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:22:50.338047 24405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:42261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:34415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:34415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:50.339128 24405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:50.340193 24405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:50.348999 24415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:50.349112 24417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:50.349073 24405 server_base.cc:1034] running on GCE node
W20260807 06:22:50.349026 24414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:50.349882 24405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:50.351123 24405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:50.352306 24405 hybrid_clock.cc:648] HybridClock initialized: now 1786083770352292 us; error 31 us; skew 500 ppm
Aug 07 06:22:50 dist-test-slave-44d2 krb5kdc[24388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083770, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:50.355366 24405 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:22:50.356366 24405 webserver.cc:460] Webserver started at http://127.15.254.62:36009/ using document root <none> and password file <none>
I20260807 06:22:50.357036 24405 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:50.357137 24405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:50.357398 24405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:50.359570 24405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "2c42d299a6b841db8fec372600e70c85"
format_stamp: "Formatted at 2026-08-07 06:22:50 on dist-test-slave-44d2"
server_key: "da5eb1696800746e14fe13216c4256ec"
server_key_iv: "06c70984df91838d296957b55194a2a6"
server_key_version: "encryptionkey@0"
I20260807 06:22:50.360167 24405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "2c42d299a6b841db8fec372600e70c85"
format_stamp: "Formatted at 2026-08-07 06:22:50 on dist-test-slave-44d2"
server_key: "da5eb1696800746e14fe13216c4256ec"
server_key_iv: "06c70984df91838d296957b55194a2a6"
server_key_version: "encryptionkey@0"
I20260807 06:22:50.364101 24405 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260807 06:22:50.366660 24427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:50.367909 24405 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 06:22:50.368062 24405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "2c42d299a6b841db8fec372600e70c85"
format_stamp: "Formatted at 2026-08-07 06:22:50 on dist-test-slave-44d2"
server_key: "da5eb1696800746e14fe13216c4256ec"
server_key_iv: "06c70984df91838d296957b55194a2a6"
server_key_version: "encryptionkey@0"
I20260807 06:22:50.368176 24405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:50.386606 24405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:50.389596 24405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:50.389822 24405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:50.397372 24405 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:34415
I20260807 06:22:50.397444 24501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:34415 every 8 connection(s)
I20260807 06:22:50.398422 24405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:22:50.401400 24503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:50.407444 24503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c42d299a6b841db8fec372600e70c85: Bootstrap starting.
I20260807 06:22:50.407605 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 24405
I20260807 06:22:50.407761 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:22:50.408054 16376 external_mini_cluster.cc:1428] Setting key f0749b43422a5e443ed4390b46687cc6
I20260807 06:22:50.410259 24503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c42d299a6b841db8fec372600e70c85: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:50.411046 24503 log.cc:824] T 00000000000000000000000000000000 P 2c42d299a6b841db8fec372600e70c85: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:50.413255 24503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c42d299a6b841db8fec372600e70c85: No bootstrap required, opened a new log
Aug 07 06:22:50 dist-test-slave-44d2 krb5kdc[24388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:50.417838 24503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c42d299a6b841db8fec372600e70c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c42d299a6b841db8fec372600e70c85" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 34415 } }
I20260807 06:22:50.418130 24503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c42d299a6b841db8fec372600e70c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:50.418200 24503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c42d299a6b841db8fec372600e70c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c42d299a6b841db8fec372600e70c85, State: Initialized, Role: FOLLOWER
I20260807 06:22:50.418727 24503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c42d299a6b841db8fec372600e70c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c42d299a6b841db8fec372600e70c85" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 34415 } }
I20260807 06:22:50.418918 24503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c42d299a6b841db8fec372600e70c85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:22:50.418988 24503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c42d299a6b841db8fec372600e70c85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:22:50.419119 24503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c42d299a6b841db8fec372600e70c85 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:50.420008 24503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c42d299a6b841db8fec372600e70c85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c42d299a6b841db8fec372600e70c85" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 34415 } }
I20260807 06:22:50.420359 24503 leader_election.cc:302] T 00000000000000000000000000000000 P 2c42d299a6b841db8fec372600e70c85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c42d299a6b841db8fec372600e70c85; no voters:
I20260807 06:22:50.420688 24503 leader_election.cc:288] T 00000000000000000000000000000000 P 2c42d299a6b841db8fec372600e70c85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:22:50.420832 24508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c42d299a6b841db8fec372600e70c85 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:50.421131 24508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c42d299a6b841db8fec372600e70c85 [term 1 LEADER]: Becoming Leader. State: Replica: 2c42d299a6b841db8fec372600e70c85, State: Running, Role: LEADER
I20260807 06:22:50.421510 24508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c42d299a6b841db8fec372600e70c85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c42d299a6b841db8fec372600e70c85" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 34415 } }
I20260807 06:22:50.422106 24503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c42d299a6b841db8fec372600e70c85 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:22:50.423542 24510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c42d299a6b841db8fec372600e70c85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c42d299a6b841db8fec372600e70c85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c42d299a6b841db8fec372600e70c85" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 34415 } } }
I20260807 06:22:50.423702 24510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c42d299a6b841db8fec372600e70c85 [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:50.423956 24506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:50.409493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35890 (local address 127.15.254.62:34415)
0807 06:22:50.409911 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:50.409928 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:50.409965 (+ 37us) server_negotiation.cc:407] Connection header received
0807 06:22:50.410667 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:50.410686 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:50.411024 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:50.411353 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:50.412206 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:50.413299 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:50.414469 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:50.414951 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:50.415072 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:50.418026 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:50.418032 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:50.418047 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:50.418080 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:50.420290 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:50.420928 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:50.420937 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:50.420945 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:50.421090 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:50.421417 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:50.421420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:50.421434 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:50.421950 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:50.422172 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:50.422500 (+ 328us) server_negotiation.cc:299] Negotiation successful
0807 06:22:50.422819 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":133,"threads_started":1}
I20260807 06:22:50.424180 24512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c42d299a6b841db8fec372600e70c85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c42d299a6b841db8fec372600e70c85. Latest consensus state: current_term: 1 leader_uuid: "2c42d299a6b841db8fec372600e70c85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c42d299a6b841db8fec372600e70c85" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 34415 } } }
I20260807 06:22:50.424278 24512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c42d299a6b841db8fec372600e70c85 [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:50.424817 24524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:22:50.428653 24524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:22:50.435002 24524 catalog_manager.cc:1347] Generated new cluster ID: 7c06993a1ff54e3db29dfbc82355fca1
I20260807 06:22:50.435093 24524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:22:50.477306 24524 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:22:50.478668 24524 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:22:50.485404 24524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c42d299a6b841db8fec372600e70c85: Generated new TSK 0
I20260807 06:22:50.486249 24524 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
I20260807 06:22:50.519969 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:0
--local_ip_for_outbound_sockets=127.15.254.1
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:34415
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:42261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:50.638185 24540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:50.638463 24540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:50.638514 24540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:50.642657 24540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:50.642719 24540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:50.642807 24540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:22:50.650663 24540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:42261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:34415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:50.652068 24540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:50.653329 24540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:50.663326 24548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:50.663380 24540 server_base.cc:1034] running on GCE node
W20260807 06:22:50.663344 24546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:50.663341 24550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:50.663966 24540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:50.664692 24540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:50.665861 24540 hybrid_clock.cc:648] HybridClock initialized: now 1786083770665840 us; error 26 us; skew 500 ppm
Aug 07 06:22:50 dist-test-slave-44d2 krb5kdc[24388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083770, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:50.668748 24540 init.cc:377] Logged in from keytab as kudu/127.15.254.1@KRBTEST.COM (short username kudu)
I20260807 06:22:50.669679 24540 webserver.cc:460] Webserver started at http://127.15.254.1:46235/ using document root <none> and password file <none>
I20260807 06:22:50.670279 24540 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:50.670344 24540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:50.670604 24540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:50.673106 24540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-0/data/instance:
uuid: "1c3d6fb05b2045bdb823564ec4da2309"
format_stamp: "Formatted at 2026-08-07 06:22:50 on dist-test-slave-44d2"
server_key: "722d96b746a01c167b6b8d6289b89b33"
server_key_iv: "5c2c025db6262226a86d077f90098d44"
server_key_version: "encryptionkey@0"
I20260807 06:22:50.673722 24540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance:
uuid: "1c3d6fb05b2045bdb823564ec4da2309"
format_stamp: "Formatted at 2026-08-07 06:22:50 on dist-test-slave-44d2"
server_key: "722d96b746a01c167b6b8d6289b89b33"
server_key_iv: "5c2c025db6262226a86d077f90098d44"
server_key_version: "encryptionkey@0"
I20260807 06:22:50.677629 24540 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:22:50.680114 24561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:50.681479 24540 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 06:22:50.681618 24540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "1c3d6fb05b2045bdb823564ec4da2309"
format_stamp: "Formatted at 2026-08-07 06:22:50 on dist-test-slave-44d2"
server_key: "722d96b746a01c167b6b8d6289b89b33"
server_key_iv: "5c2c025db6262226a86d077f90098d44"
server_key_version: "encryptionkey@0"
I20260807 06:22:50.681727 24540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:50.690691 24540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:50.693672 24540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:50.693881 24540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:50.694484 24540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:50.695328 24540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:50.695374 24540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:50.695444 24540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:50.695477 24540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:50.705125 24540 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:36323
I20260807 06:22:50.705156 24722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:36323 every 8 connection(s)
I20260807 06:22:50.706081 24540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:22:50.708350 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 24540
I20260807 06:22:50.708515 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance
I20260807 06:22:50.708874 16376 external_mini_cluster.cc:1428] Setting key 5807bc9d6c8a363c5141a748a392b119
Aug 07 06:22:50 dist-test-slave-44d2 krb5kdc[24388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083770, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.254.2@KRBTEST.COM; defaulting to no policy
I20260807 06:22:50.722723 24506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:50.708003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:41847 (local address 127.15.254.62:34415)
0807 06:22:50.708224 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:50.708230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:50.709300 (+ 1070us) server_negotiation.cc:407] Connection header received
0807 06:22:50.710925 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:50.710931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:50.710999 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:50.711096 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:50.712424 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:50.713142 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:50.715013 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:50.715243 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:50.715322 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:50.718629 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:50.718633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:50.718635 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:50.718660 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:50.720225 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:50.720886 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:50.720890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:50.720892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:50.720981 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:50.721388 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:50.721392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:50.721394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:50.721666 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:50.721780 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:50.722320 (+ 540us) server_negotiation.cc:299] Negotiation successful
0807 06:22:50.722458 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.2@KRBTEST.COM" created.
I20260807 06:22:50.723793 24727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:50.708409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34415 (local address 127.15.254.1:41847)
0807 06:22:50.709112 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:50.709175 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:50.710693 (+ 1518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:50.711291 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:50.711298 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:50.711688 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:50.712246 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:50.712264 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:50.713299 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:50.713305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:50.714869 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:50.714885 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:50.715416 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:50.715423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:50.715683 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:50.716535 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:50.716560 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:50.718432 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:50.720366 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:50.720372 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:50.720390 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:50.720773 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:50.721111 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:50.721115 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:50.721117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:50.721276 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:50.721779 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:50.721787 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:50.722063 (+ 276us) client_negotiation.cc:769] Sending connection context
0807 06:22:50.722261 (+ 198us) client_negotiation.cc:240] Negotiation successful
0807 06:22:50.722637 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":146,"threads_started":1}
I20260807 06:22:50.725428 24724 heartbeater.cc:347] Connected to a master server at 127.15.254.62:34415
I20260807 06:22:50.725713 24724 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:50.726411 24724 heartbeater.cc:511] Master 127.15.254.62:34415 requested a full tablet report, sending...
I20260807 06:22:50.728109 24455 ts_manager.cc:194] Registered new tserver with Master: 1c3d6fb05b2045bdb823564ec4da2309 (127.15.254.1:36323)
I20260807 06:22:50.729394 24455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.1@KRBTEST.COM'} at 127.15.254.1:41847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
I20260807 06:22:50.743546 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:0
--local_ip_for_outbound_sockets=127.15.254.2
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:34415
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:42261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:50.861828 24730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:50.862082 24730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:50.862169 24730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:50.866297 24730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:50.866349 24730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:50.866452 24730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:22:50.871824 24730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:42261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:34415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:50.873078 24730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:50.874240 24730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:50.884666 24740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:50.884653 24736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:50.884677 24737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:50.884917 24730 server_base.cc:1034] running on GCE node
I20260807 06:22:50.885358 24730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:50.886147 24730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:50.887392 24730 hybrid_clock.cc:648] HybridClock initialized: now 1786083770887357 us; error 59 us; skew 500 ppm
Aug 07 06:22:50 dist-test-slave-44d2 krb5kdc[24388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083770, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:50.890919 24730 init.cc:377] Logged in from keytab as kudu/127.15.254.2@KRBTEST.COM (short username kudu)
I20260807 06:22:50.891881 24730 webserver.cc:460] Webserver started at http://127.15.254.2:37177/ using document root <none> and password file <none>
I20260807 06:22:50.892567 24730 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:50.892650 24730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:50.892895 24730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:50.895027 24730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-1/data/instance:
uuid: "68f906efb99d473bb59c908358e9bad6"
format_stamp: "Formatted at 2026-08-07 06:22:50 on dist-test-slave-44d2"
server_key: "5955bc79a1958ee99bea8474c3752dc3"
server_key_iv: "3a2fdbfaaedea2d81f7256013a369581"
server_key_version: "encryptionkey@0"
I20260807 06:22:50.895619 24730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance:
uuid: "68f906efb99d473bb59c908358e9bad6"
format_stamp: "Formatted at 2026-08-07 06:22:50 on dist-test-slave-44d2"
server_key: "5955bc79a1958ee99bea8474c3752dc3"
server_key_iv: "3a2fdbfaaedea2d81f7256013a369581"
server_key_version: "encryptionkey@0"
I20260807 06:22:50.899537 24730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260807 06:22:50.902143 24748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:50.903345 24730 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 06:22:50.903491 24730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "68f906efb99d473bb59c908358e9bad6"
format_stamp: "Formatted at 2026-08-07 06:22:50 on dist-test-slave-44d2"
server_key: "5955bc79a1958ee99bea8474c3752dc3"
server_key_iv: "3a2fdbfaaedea2d81f7256013a369581"
server_key_version: "encryptionkey@0"
I20260807 06:22:50.903599 24730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:50.960733 24730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:50.964139 24730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:50.964363 24730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:50.965057 24730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:50.965986 24730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:50.966035 24730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:50.966086 24730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:50.966144 24730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:50.976256 24730 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:36749
I20260807 06:22:50.976284 24913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:36749 every 8 connection(s)
I20260807 06:22:50.977250 24730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:22:50.982153 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 24730
I20260807 06:22:50.982295 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance
I20260807 06:22:50.982599 16376 external_mini_cluster.cc:1428] Setting key 737f96538bbfa4c3b1c0ae5ee95f07e9
Aug 07 06:22:50 dist-test-slave-44d2 krb5kdc[24388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083770, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:50.992997 24506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:50.979252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:50049 (local address 127.15.254.62:34415)
0807 06:22:50.979435 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:50.979440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:50.980306 (+ 866us) server_negotiation.cc:407] Connection header received
0807 06:22:50.981728 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:50.981731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:50.981786 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:50.981863 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:50.983225 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:50.983870 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:50.985511 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:50.985926 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:50.986060 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:50.989040 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:50.989043 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:50.989046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:50.989072 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:50.990643 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:50.991226 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:50.991230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:50.991231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:50.991315 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:50.991703 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:50.991706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:50.991708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:50.991914 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:50.992074 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:50.992670 (+ 596us) server_negotiation.cc:299] Negotiation successful
0807 06:22:50.992813 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
WARNING: no policy specified for kudu/127.15.254.3@KRBTEST.COM; defaulting to no policy
I20260807 06:22:50.993950 24916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:50.979658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34415 (local address 127.15.254.2:50049)
0807 06:22:50.980137 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:50.980183 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:50.981507 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:50.982056 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:50.982063 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:50.982526 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:50.983090 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:50.983105 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:50.984020 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:50.984024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:50.985385 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:50.985397 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:50.986195 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:50.986198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:50.986406 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:50.987103 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:50.987129 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:50.988847 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:50.990780 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:50.990788 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:50.990806 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:50.991098 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:50.991417 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:50.991421 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:50.991423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:50.991593 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:50.992031 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:50.992040 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:50.992340 (+ 300us) client_negotiation.cc:769] Sending connection context
0807 06:22:50.992602 (+ 262us) client_negotiation.cc:240] Negotiation successful
0807 06:22:50.992990 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":124,"threads_started":1}
I20260807 06:22:50.995256 24914 heartbeater.cc:347] Connected to a master server at 127.15.254.62:34415
I20260807 06:22:50.995494 24914 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:50.995992 24914 heartbeater.cc:511] Master 127.15.254.62:34415 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.3@KRBTEST.COM" created.
I20260807 06:22:50.997262 24455 ts_manager.cc:194] Registered new tserver with Master: 68f906efb99d473bb59c908358e9bad6 (127.15.254.2:36749)
I20260807 06:22:50.997921 24455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.2@KRBTEST.COM'} at 127.15.254.2:50049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
I20260807 06:22:51.016829 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:0
--local_ip_for_outbound_sockets=127.15.254.3
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:34415
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:42261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:51.143368 24919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:51.143785 24919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:51.143918 24919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:51.148240 24919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:51.148340 24919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:51.148438 24919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:22:51.154007 24919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:42261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:34415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:51.155261 24919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:51.156433 24919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:51.167066 24925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:51.167089 24924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:51.167074 24927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:51.167399 24919 server_base.cc:1034] running on GCE node
I20260807 06:22:51.167884 24919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:51.168638 24919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:51.169812 24919 hybrid_clock.cc:648] HybridClock initialized: now 1786083771169779 us; error 42 us; skew 500 ppm
Aug 07 06:22:51 dist-test-slave-44d2 krb5kdc[24388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083771, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:51.176031 24919 init.cc:377] Logged in from keytab as kudu/127.15.254.3@KRBTEST.COM (short username kudu)
I20260807 06:22:51.177094 24919 webserver.cc:460] Webserver started at http://127.15.254.3:39467/ using document root <none> and password file <none>
I20260807 06:22:51.177701 24919 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:51.177769 24919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:51.178025 24919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:51.180284 24919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-2/data/instance:
uuid: "69282a778c9e4c0d85e478b456a86f36"
format_stamp: "Formatted at 2026-08-07 06:22:51 on dist-test-slave-44d2"
server_key: "b04303bd26c039f7f4e6b9e768015221"
server_key_iv: "ba404de7db928911e1cc92f4887e35e0"
server_key_version: "encryptionkey@0"
I20260807 06:22:51.180935 24919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance:
uuid: "69282a778c9e4c0d85e478b456a86f36"
format_stamp: "Formatted at 2026-08-07 06:22:51 on dist-test-slave-44d2"
server_key: "b04303bd26c039f7f4e6b9e768015221"
server_key_iv: "ba404de7db928911e1cc92f4887e35e0"
server_key_version: "encryptionkey@0"
I20260807 06:22:51.185029 24919 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:22:51.187906 24940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:51.189373 24919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:22:51.189518 24919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "69282a778c9e4c0d85e478b456a86f36"
format_stamp: "Formatted at 2026-08-07 06:22:51 on dist-test-slave-44d2"
server_key: "b04303bd26c039f7f4e6b9e768015221"
server_key_iv: "ba404de7db928911e1cc92f4887e35e0"
server_key_version: "encryptionkey@0"
I20260807 06:22:51.189635 24919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:51.213317 24919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:51.216573 24919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:51.216791 24919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:51.217479 24919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:51.218410 24919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:51.218459 24919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:51.218536 24919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:51.218575 24919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:51.228148 24919 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:39027
I20260807 06:22:51.228268 25100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:39027 every 8 connection(s)
I20260807 06:22:51.229205 24919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:22:51.235395 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 24919
I20260807 06:22:51.235555 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance
I20260807 06:22:51.235880 16376 external_mini_cluster.cc:1428] Setting key 9a6929970cea13dddecc93cd422b780b
Aug 07 06:22:51 dist-test-slave-44d2 krb5kdc[24388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083771, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:22:51.244839 24506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:51.231246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:41111 (local address 127.15.254.62:34415)
0807 06:22:51.231399 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:51.231403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:51.232328 (+ 925us) server_negotiation.cc:407] Connection header received
0807 06:22:51.233829 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:51.233832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:51.233887 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:51.233979 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:51.235289 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:51.236087 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:51.237960 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:51.238185 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:51.238277 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:51.241022 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:51.241026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:51.241031 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:51.241062 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:51.242636 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:51.243199 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:51.243203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:51.243204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:51.243283 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:51.243648 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:51.243651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:51.243652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:51.243839 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:51.243951 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:51.244473 (+ 522us) server_negotiation.cc:299] Negotiation successful
0807 06:22:51.244659 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 06:22:51.245651 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:51.231647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34415 (local address 127.15.254.3:41111)
0807 06:22:51.232147 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:51.232195 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:51.233593 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:51.234177 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:51.234184 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:51.234582 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:51.235129 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:51.235143 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:51.236243 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:51.236246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:51.237808 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:51.237822 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:51.238354 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:51.238358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:51.238569 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:51.239193 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:51.239211 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:51.240840 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:51.242770 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:51.242776 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:51.242792 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:51.243076 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:51.243379 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:51.243383 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:51.243385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:51.243538 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:51.243951 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:51.243959 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:51.244220 (+ 261us) client_negotiation.cc:769] Sending connection context
0807 06:22:51.244414 (+ 194us) client_negotiation.cc:240] Negotiation successful
0807 06:22:51.244785 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":138,"threads_started":1}
I20260807 06:22:51.246790 25101 heartbeater.cc:347] Connected to a master server at 127.15.254.62:34415
I20260807 06:22:51.247058 25101 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:51.247582 25101 heartbeater.cc:511] Master 127.15.254.62:34415 requested a full tablet report, sending...
I20260807 06:22:51.248692 24455 ts_manager.cc:194] Registered new tserver with Master: 69282a778c9e4c0d85e478b456a86f36 (127.15.254.3:39027)
I20260807 06:22:51.249358 24455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.3@KRBTEST.COM'} at 127.15.254.3:41111
I20260807 06:22:51.250612 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/test-admin.keytab.
Aug 07 06:22:51 dist-test-slave-44d2 krb5kdc[24388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:51.270009 16376 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260807 06:22:51.273674 25109 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:22:51.271780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43630 (local address 127.15.254.1:36323)
0807 06:22:51.272239 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:51.272246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:51.272272 (+ 26us) server_negotiation.cc:407] Connection header received
0807 06:22:51.272327 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:51.272332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:51.272557 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:51.272709 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:51.273494 (+ 785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43630: BlockingRecv error: recv got EOF from 127.0.0.1:43630 (error 108)
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":198,"threads_started":1}
W20260807 06:22:51.274073 16376 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/krb5cc' of type FILE
Aug 07 06:22:51 dist-test-slave-44d2 krb5kdc[24388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:51.275285 16376 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 06:22:51 dist-test-slave-44d2 krb5kdc[24388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:51.276355 16376 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 06:22:51 dist-test-slave-44d2 krb5kdc[24388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.1@KRBTEST.COM
I20260807 06:22:51.286939 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:51.277440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43644 (local address 127.15.254.1:36323)
0807 06:22:51.277583 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:51.277587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:51.277602 (+ 15us) server_negotiation.cc:407] Connection header received
0807 06:22:51.277703 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:51.277706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:51.277750 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:51.277819 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:51.278412 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:51.279201 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:51.280038 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:51.280420 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:51.280622 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:51.283061 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:51.283084 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:51.283105 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:51.283132 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:51.285025 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:51.285464 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:51.285469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:51.285470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:51.285622 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:51.285888 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:51.285890 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:51.285891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:51.286228 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:51.286390 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:51.286580 (+ 190us) server_negotiation.cc:299] Negotiation successful
0807 06:22:51.286713 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260807 06:22:51.290807 25109 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:22:51.289923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43658 (local address 127.15.254.1:36323)
0807 06:22:51.290082 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:51.290085 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:51.290098 (+ 13us) server_negotiation.cc:407] Connection header received
0807 06:22:51.290202 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:51.290204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:51.290241 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:51.290305 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:51.290716 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43658: BlockingRecv error: recv got EOF from 127.0.0.1:43658 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260807 06:22:51.290920 16376 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/krb5cc' of type FILE
Aug 07 06:22:51 dist-test-slave-44d2 krb5kdc[24388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:51.291989 16376 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 06:22:51 dist-test-slave-44d2 krb5kdc[24388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:51.293098 16376 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 06:22:51 dist-test-slave-44d2 krb5kdc[24388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.1@KRBTEST.COM
I20260807 06:22:51.302042 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:51.293951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43662 (local address 127.15.254.1:36323)
0807 06:22:51.294068 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:51.294072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:51.294246 (+ 174us) server_negotiation.cc:407] Connection header received
0807 06:22:51.294377 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:51.294381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:51.294425 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:51.294516 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:51.295077 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:51.295575 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:51.296389 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:51.296715 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:51.296831 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:51.299085 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:51.299093 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:51.299098 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:51.299125 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:51.300355 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:51.300814 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:51.300818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:51.300819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:51.300904 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:51.301191 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:51.301195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:51.301196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:51.301373 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:51.301540 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:51.301769 (+ 229us) server_negotiation.cc:299] Negotiation successful
0807 06:22:51.301888 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260807 06:22:51.304708 25109 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:22:51.303771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43664 (local address 127.15.254.1:36323)
0807 06:22:51.303911 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:51.303915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:51.303942 (+ 27us) server_negotiation.cc:407] Connection header received
0807 06:22:51.304036 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:51.304039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:51.304093 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:51.304152 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:51.304590 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43664: BlockingRecv error: recv got EOF from 127.0.0.1:43664 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260807 06:22:51.304752 16376 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786083738959718-16376-0/krb5kdc/krb5cc' of type FILE
Aug 07 06:22:51 dist-test-slave-44d2 krb5kdc[24388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:51.305747 16376 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 06:22:51 dist-test-slave-44d2 krb5kdc[24388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:51.306636 16376 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 06:22:51 dist-test-slave-44d2 krb5kdc[24388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.1@KRBTEST.COM
I20260807 06:22:51.315543 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:51.307515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43672 (local address 127.15.254.1:36323)
0807 06:22:51.307651 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:51.307654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:51.307693 (+ 39us) server_negotiation.cc:407] Connection header received
0807 06:22:51.307805 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:51.307807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:51.307842 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:51.307906 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:51.308401 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:51.308969 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:51.309727 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:51.309934 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:51.310012 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:51.312298 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:51.312302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:51.312304 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:51.312331 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:51.313910 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:51.314358 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:51.314360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:51.314362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:51.314439 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:51.314695 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:51.314697 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:51.314698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:51.314871 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:51.314993 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:51.315211 (+ 218us) server_negotiation.cc:299] Negotiation successful
0807 06:22:51.315333 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 06:22:51.316923 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 24540
I20260807 06:22:51.333762 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 24730
I20260807 06:22:51.343979 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 24919
I20260807 06:22:51.354049 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 24405
2026-08-07T06:22:51Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2520 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:22:51 dist-test-slave-44d2 krb5kdc[25128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:22:51 dist-test-slave-44d2 krb5kdc[25128](info): set up 2 sockets
Aug 07 06:22:51 dist-test-slave-44d2 krb5kdc[25128](info): commencing operation
krb5kdc: starting...
W20260807 06:22:52.601624 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.218s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:22:52 dist-test-slave-44d2 krb5kdc[25128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:22:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:22:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.62.keytab.
Entry for principal oryx/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.62.keytab.
I20260807 06:22:52.740069 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:41399
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.62.keytab
--principal=oryx/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:41399
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:52.863456 25149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:52.863790 25149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:52.863888 25149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:52.867990 25149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:22:52.868081 25149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:52.868119 25149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:52.868180 25149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:22:52.868237 25149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:22:52.873999 25149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:41399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.62.keytab
--principal=oryx/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:41399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:52.875216 25149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:52.876426 25149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:52.885756 25160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:52.885771 25162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:52.885854 25157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:52.886040 25149 server_base.cc:1034] running on GCE node
I20260807 06:22:52.886613 25149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:52.887778 25149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:52.889010 25149 hybrid_clock.cc:648] HybridClock initialized: now 1786083772888989 us; error 33 us; skew 500 ppm
Aug 07 06:22:52 dist-test-slave-44d2 krb5kdc[25128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083772, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:52.891898 25149 init.cc:377] Logged in from keytab as oryx/127.15.254.62@KRBTEST.COM (short username oryx)
I20260807 06:22:52.892976 25149 webserver.cc:460] Webserver started at http://127.15.254.62:39105/ using document root <none> and password file <none>
I20260807 06:22:52.893568 25149 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:52.893667 25149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:52.894019 25149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:52.896742 25149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "9f97442df1a34ff6a8a6215460b56d62"
format_stamp: "Formatted at 2026-08-07 06:22:52 on dist-test-slave-44d2"
server_key: "688f3fa987222cda0cc4752d96c36c74"
server_key_iv: "62f4e1ee1f4544722cddef5deecaaec7"
server_key_version: "encryptionkey@0"
I20260807 06:22:52.897467 25149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "9f97442df1a34ff6a8a6215460b56d62"
format_stamp: "Formatted at 2026-08-07 06:22:52 on dist-test-slave-44d2"
server_key: "688f3fa987222cda0cc4752d96c36c74"
server_key_iv: "62f4e1ee1f4544722cddef5deecaaec7"
server_key_version: "encryptionkey@0"
I20260807 06:22:52.901373 25149 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:22:52.903791 25175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:52.905102 25149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260807 06:22:52.905251 25149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "9f97442df1a34ff6a8a6215460b56d62"
format_stamp: "Formatted at 2026-08-07 06:22:52 on dist-test-slave-44d2"
server_key: "688f3fa987222cda0cc4752d96c36c74"
server_key_iv: "62f4e1ee1f4544722cddef5deecaaec7"
server_key_version: "encryptionkey@0"
I20260807 06:22:52.905364 25149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:52.928905 25149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:52.932085 25149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:52.932315 25149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:52.940161 25149 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:41399
I20260807 06:22:52.940517 25244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:41399 every 8 connection(s)
I20260807 06:22:52.941254 25149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:22:52.944175 25245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:52.947098 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 25149
I20260807 06:22:52.947243 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:22:52.947566 16376 external_mini_cluster.cc:1428] Setting key 42a51583ad0806f026ee5f07bce9465e
I20260807 06:22:52.950613 25245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62: Bootstrap starting.
I20260807 06:22:52.953552 25245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:52.954528 25245 log.cc:824] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:52.957031 25245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62: No bootstrap required, opened a new log
Aug 07 06:22:52 dist-test-slave-44d2 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM
I20260807 06:22:52.961822 25245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } }
I20260807 06:22:52.962162 25245 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:52.962219 25245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f97442df1a34ff6a8a6215460b56d62, State: Initialized, Role: FOLLOWER
I20260807 06:22:52.962756 25245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } }
I20260807 06:22:52.962941 25245 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:22:52.963016 25245 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:22:52.963179 25245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:52.964233 25245 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } }
I20260807 06:22:52.964466 25249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:52.949033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47976 (local address 127.15.254.62:41399)
0807 06:22:52.949494 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:52.949511 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:52.949548 (+ 37us) server_negotiation.cc:407] Connection header received
0807 06:22:52.950193 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:52.950212 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:52.950569 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:52.950904 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:52.951826 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:52.952978 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:52.954204 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:52.954779 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:52.954911 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:52.958125 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:52.958140 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:52.958160 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:52.958196 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:52.961001 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:52.961551 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:52.961557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:52.961570 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:52.961725 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:52.962045 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:52.962048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:52.962050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:52.962659 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:52.962880 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:52.963244 (+ 364us) server_negotiation.cc:299] Negotiation successful
0807 06:22:52.963552 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":148,"threads_started":1}
I20260807 06:22:52.964702 25245 leader_election.cc:302] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f97442df1a34ff6a8a6215460b56d62; no voters:
I20260807 06:22:52.965088 25245 leader_election.cc:288] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:22:52.965193 25251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:52.965494 25251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [term 1 LEADER]: Becoming Leader. State: Replica: 9f97442df1a34ff6a8a6215460b56d62, State: Running, Role: LEADER
I20260807 06:22:52.965879 25251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } }
I20260807 06:22:52.966423 25245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:22:52.967103 25252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f97442df1a34ff6a8a6215460b56d62. Latest consensus state: current_term: 1 leader_uuid: "9f97442df1a34ff6a8a6215460b56d62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } }
I20260807 06:22:52.967338 25252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:52.967607 25253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f97442df1a34ff6a8a6215460b56d62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } }
I20260807 06:22:52.967717 25253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:52.967834 25260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:22:52.971027 25260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:22:52.977216 25260 catalog_manager.cc:1347] Generated new cluster ID: b67ecb0da3614575836bf6cf3f400654
I20260807 06:22:52.977289 25260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:22:52.983755 25260 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:22:52.984920 25260 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:22:52.997257 25260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62: Generated new TSK 0
I20260807 06:22:52.998241 25260 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.1.keytab.
Entry for principal oryx/127.15.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.1.keytab.
I20260807 06:22:53.040182 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:0
--local_ip_for_outbound_sockets=127.15.254.1
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:41399
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.1.keytab
--principal=oryx/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:53.158649 25273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:53.158934 25273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:53.158986 25273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:53.163112 25273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:53.163189 25273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:53.163256 25273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:22:53.168963 25273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.1.keytab
--principal=oryx/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:41399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:53.169999 25273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:53.171001 25273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:53.181176 25280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:53.181185 25281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:53.181293 25273 server_base.cc:1034] running on GCE node
W20260807 06:22:53.181191 25283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:53.181829 25273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:53.182453 25273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:53.183620 25273 hybrid_clock.cc:648] HybridClock initialized: now 1786083773183598 us; error 35 us; skew 500 ppm
Aug 07 06:22:53 dist-test-slave-44d2 krb5kdc[25128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083773, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:53.186733 25273 init.cc:377] Logged in from keytab as oryx/127.15.254.1@KRBTEST.COM (short username oryx)
I20260807 06:22:53.187677 25273 webserver.cc:460] Webserver started at http://127.15.254.1:37981/ using document root <none> and password file <none>
I20260807 06:22:53.188264 25273 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:53.188329 25273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:53.188653 25273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:53.190904 25273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/data/instance:
uuid: "dc9fbde76a594443bf8b835f3b566d63"
format_stamp: "Formatted at 2026-08-07 06:22:53 on dist-test-slave-44d2"
server_key: "133d5441ba54cbd2b3c587b8effad3c1"
server_key_iv: "5e52ed9d1d9efc88f5ed35d09690b7ae"
server_key_version: "encryptionkey@0"
I20260807 06:22:53.191514 25273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance:
uuid: "dc9fbde76a594443bf8b835f3b566d63"
format_stamp: "Formatted at 2026-08-07 06:22:53 on dist-test-slave-44d2"
server_key: "133d5441ba54cbd2b3c587b8effad3c1"
server_key_iv: "5e52ed9d1d9efc88f5ed35d09690b7ae"
server_key_version: "encryptionkey@0"
I20260807 06:22:53.195725 25273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 06:22:53.198685 25293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:53.200152 25273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:22:53.200346 25273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "dc9fbde76a594443bf8b835f3b566d63"
format_stamp: "Formatted at 2026-08-07 06:22:53 on dist-test-slave-44d2"
server_key: "133d5441ba54cbd2b3c587b8effad3c1"
server_key_iv: "5e52ed9d1d9efc88f5ed35d09690b7ae"
server_key_version: "encryptionkey@0"
I20260807 06:22:53.200532 25273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:53.213735 25273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:53.217438 25273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:53.217670 25273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:53.218314 25273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:53.219223 25273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:53.219272 25273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:53.219352 25273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:53.219384 25273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:53.229629 25273 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:42279
I20260807 06:22:53.229652 25456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:42279 every 8 connection(s)
I20260807 06:22:53.230553 25273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:22:53.238627 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 25273
I20260807 06:22:53.238785 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance
I20260807 06:22:53.239190 16376 external_mini_cluster.cc:1428] Setting key 39177e6b907ee1f899efad92c5d0f9eb
Aug 07 06:22:53 dist-test-slave-44d2 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083773, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM
I20260807 06:22:53.246048 25249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:53.232612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:51229 (local address 127.15.254.62:41399)
0807 06:22:53.232787 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:53.232793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:53.233812 (+ 1019us) server_negotiation.cc:407] Connection header received
0807 06:22:53.235240 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:53.235244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:53.235303 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:53.235384 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:53.236723 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:53.237304 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:53.238594 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:53.238874 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:53.239034 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:53.242043 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:53.242050 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:53.242056 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:53.242092 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:53.243856 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:53.244395 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:53.244398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:53.244400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:53.244543 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:53.244916 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:53.244919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:53.244921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:53.245110 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:53.245217 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:53.245738 (+ 521us) server_negotiation.cc:299] Negotiation successful
0807 06:22:53.245870 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 06:22:53.246876 25459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:53.233121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.1:51229)
0807 06:22:53.233641 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:53.233688 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:53.235025 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:53.235598 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:53.235605 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:53.236036 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:53.236580 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:53.236596 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:53.237431 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:53.237434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:53.238456 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:53.238468 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:53.239094 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:53.239098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:53.239307 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:53.240102 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:53.240122 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:53.241852 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:53.243982 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:53.243989 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:53.244004 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:53.244280 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:53.244647 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:53.244651 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:53.244653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:53.244806 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:53.245232 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:53.245240 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:53.245490 (+ 250us) client_negotiation.cc:769] Sending connection context
0807 06:22:53.245682 (+ 192us) client_negotiation.cc:240] Negotiation successful
0807 06:22:53.246031 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":118,"threads_started":1}
I20260807 06:22:53.248566 25457 heartbeater.cc:347] Connected to a master server at 127.15.254.62:41399
I20260807 06:22:53.248827 25457 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:53.249402 25457 heartbeater.cc:511] Master 127.15.254.62:41399 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.15.254.2@KRBTEST.COM; defaulting to no policy
I20260807 06:22:53.251267 25193 ts_manager.cc:194] Registered new tserver with Master: dc9fbde76a594443bf8b835f3b566d63 (127.15.254.1:42279)
I20260807 06:22:53.252640 25193 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.254.1@KRBTEST.COM'} at 127.15.254.1:51229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.254.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.2.keytab.
Entry for principal oryx/127.15.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.2.keytab.
I20260807 06:22:53.272152 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:0
--local_ip_for_outbound_sockets=127.15.254.2
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:41399
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.2.keytab
--principal=oryx/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:53.391752 25464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:53.392079 25464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:53.392168 25464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:53.396382 25464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:53.396523 25464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:53.396636 25464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:22:53.402063 25464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.2.keytab
--principal=oryx/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:41399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:53.403184 25464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:53.404369 25464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:53.414681 25470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:53.414778 25464 server_base.cc:1034] running on GCE node
W20260807 06:22:53.414752 25474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:53.414944 25469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:53.415378 25464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:53.416118 25464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:53.417317 25464 hybrid_clock.cc:648] HybridClock initialized: now 1786083773417294 us; error 37 us; skew 500 ppm
Aug 07 06:22:53 dist-test-slave-44d2 krb5kdc[25128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083773, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:53.420162 25464 init.cc:377] Logged in from keytab as oryx/127.15.254.2@KRBTEST.COM (short username oryx)
I20260807 06:22:53.421150 25464 webserver.cc:460] Webserver started at http://127.15.254.2:39363/ using document root <none> and password file <none>
I20260807 06:22:53.421749 25464 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:53.421849 25464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:53.422101 25464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:53.424232 25464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/data/instance:
uuid: "f0e00ec4b6044d4ea221ea3da197ad7d"
format_stamp: "Formatted at 2026-08-07 06:22:53 on dist-test-slave-44d2"
server_key: "8950360f6d3a6df586a8320cd35917e1"
server_key_iv: "20fde2503d9912613d4631afefc9bb2e"
server_key_version: "encryptionkey@0"
I20260807 06:22:53.424896 25464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance:
uuid: "f0e00ec4b6044d4ea221ea3da197ad7d"
format_stamp: "Formatted at 2026-08-07 06:22:53 on dist-test-slave-44d2"
server_key: "8950360f6d3a6df586a8320cd35917e1"
server_key_iv: "20fde2503d9912613d4631afefc9bb2e"
server_key_version: "encryptionkey@0"
I20260807 06:22:53.428808 25464 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:22:53.431375 25484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:53.432787 25464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:22:53.432933 25464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "f0e00ec4b6044d4ea221ea3da197ad7d"
format_stamp: "Formatted at 2026-08-07 06:22:53 on dist-test-slave-44d2"
server_key: "8950360f6d3a6df586a8320cd35917e1"
server_key_iv: "20fde2503d9912613d4631afefc9bb2e"
server_key_version: "encryptionkey@0"
I20260807 06:22:53.433058 25464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:53.450498 25464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:53.454210 25464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:53.454438 25464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:53.455096 25464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:53.455999 25464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:53.456086 25464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:53.456157 25464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:53.456188 25464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:53.466037 25464 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:34939
I20260807 06:22:53.466073 25645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:34939 every 8 connection(s)
I20260807 06:22:53.467075 25464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:22:53.470475 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 25464
I20260807 06:22:53.470594 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance
I20260807 06:22:53.471019 16376 external_mini_cluster.cc:1428] Setting key a37a1c25471047dfac821826f9733dcb
Aug 07 06:22:53 dist-test-slave-44d2 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083773, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.15.254.3@KRBTEST.COM; defaulting to no policy
I20260807 06:22:53.484467 25249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:53.469127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:49783 (local address 127.15.254.62:41399)
0807 06:22:53.469287 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:53.469292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:53.470237 (+ 945us) server_negotiation.cc:407] Connection header received
0807 06:22:53.472119 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:53.472122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:53.472173 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:53.472265 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:53.474079 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:53.474618 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:53.476112 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:53.476344 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:53.476464 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:53.479133 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:53.479137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:53.479139 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:53.479165 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:53.482115 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:53.482765 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:53.482769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:53.482770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:53.482866 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:53.483234 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:53.483238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:53.483239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:53.483448 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:53.483565 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:53.484155 (+ 590us) server_negotiation.cc:299] Negotiation successful
0807 06:22:53.484295 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 06:22:53.485483 25650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:53.469600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.2:49783)
0807 06:22:53.470072 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:53.470118 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:53.471844 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:53.472503 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:53.472517 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:53.473095 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:53.473894 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:53.473920 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:53.474765 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:53.474771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:53.475958 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:53.475969 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:53.476553 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:53.476558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:53.476743 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:53.477411 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:53.477429 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:53.478946 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:53.482263 (+ 3317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:53.482271 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:53.482288 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:53.482618 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:53.482965 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:53.482968 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:53.482969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:53.483116 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:53.483565 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:53.483573 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:53.483895 (+ 322us) client_negotiation.cc:769] Sending connection context
0807 06:22:53.484113 (+ 218us) client_negotiation.cc:240] Negotiation successful
0807 06:22:53.484473 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.254.3@KRBTEST.COM" created.
I20260807 06:22:53.487046 25646 heartbeater.cc:347] Connected to a master server at 127.15.254.62:41399
I20260807 06:22:53.487323 25646 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:53.487852 25646 heartbeater.cc:511] Master 127.15.254.62:41399 requested a full tablet report, sending...
I20260807 06:22:53.489120 25193 ts_manager.cc:194] Registered new tserver with Master: f0e00ec4b6044d4ea221ea3da197ad7d (127.15.254.2:34939)
I20260807 06:22:53.489943 25193 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.254.2@KRBTEST.COM'} at 127.15.254.2:49783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.3.keytab.
Entry for principal oryx/127.15.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.3.keytab.
I20260807 06:22:53.504235 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:0
--local_ip_for_outbound_sockets=127.15.254.3
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:41399
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.3.keytab
--principal=oryx/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:53.624265 25654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:53.624569 25654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:53.624647 25654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:53.628741 25654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:53.628823 25654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:53.628908 25654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:22:53.634325 25654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.3.keytab
--principal=oryx/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:41399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:53.635515 25654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:53.636790 25654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:53.647313 25662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:53.647359 25664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:53.647322 25668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:53.647537 25654 server_base.cc:1034] running on GCE node
I20260807 06:22:53.648141 25654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:53.648938 25654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:53.650156 25654 hybrid_clock.cc:648] HybridClock initialized: now 1786083773650122 us; error 51 us; skew 500 ppm
Aug 07 06:22:53 dist-test-slave-44d2 krb5kdc[25128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083773, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:53.653079 25654 init.cc:377] Logged in from keytab as oryx/127.15.254.3@KRBTEST.COM (short username oryx)
I20260807 06:22:53.653918 25654 webserver.cc:460] Webserver started at http://127.15.254.3:32961/ using document root <none> and password file <none>
I20260807 06:22:53.654465 25654 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:53.654527 25654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:53.654703 25654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:53.656894 25654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/data/instance:
uuid: "043fb1b788ed4af18200209e0ac9f5c0"
format_stamp: "Formatted at 2026-08-07 06:22:53 on dist-test-slave-44d2"
server_key: "5960ec65e549585d70803db126a769d6"
server_key_iv: "59100694866f66072cc34cf58f47890e"
server_key_version: "encryptionkey@0"
I20260807 06:22:53.657419 25654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance:
uuid: "043fb1b788ed4af18200209e0ac9f5c0"
format_stamp: "Formatted at 2026-08-07 06:22:53 on dist-test-slave-44d2"
server_key: "5960ec65e549585d70803db126a769d6"
server_key_iv: "59100694866f66072cc34cf58f47890e"
server_key_version: "encryptionkey@0"
I20260807 06:22:53.661229 25654 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 06:22:53.663770 25676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:53.665235 25654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 06:22:53.665354 25654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "043fb1b788ed4af18200209e0ac9f5c0"
format_stamp: "Formatted at 2026-08-07 06:22:53 on dist-test-slave-44d2"
server_key: "5960ec65e549585d70803db126a769d6"
server_key_iv: "59100694866f66072cc34cf58f47890e"
server_key_version: "encryptionkey@0"
I20260807 06:22:53.665452 25654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:53.684616 25654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:53.687660 25654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:53.687816 25654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:53.688378 25654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:53.689294 25654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:53.689343 25654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:53.689385 25654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:53.689410 25654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:53.699105 25654 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:43515
I20260807 06:22:53.699128 25841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:43515 every 8 connection(s)
I20260807 06:22:53.700069 25654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:22:53.702322 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 25654
I20260807 06:22:53.702476 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance
I20260807 06:22:53.702822 16376 external_mini_cluster.cc:1428] Setting key 734ac64fcf6372775aaa179b0c8d43fc
Aug 07 06:22:53 dist-test-slave-44d2 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083773, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM
I20260807 06:22:53.716295 25249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:53.701999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:36657 (local address 127.15.254.62:41399)
0807 06:22:53.702177 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:53.702182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:53.703234 (+ 1052us) server_negotiation.cc:407] Connection header received
0807 06:22:53.705108 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:53.705111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:53.705172 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:53.705258 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:53.706660 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:53.707192 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:53.708546 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:53.708778 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:53.708862 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:53.712292 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:53.712296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:53.712299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:53.712327 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:53.713724 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:53.714411 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:53.714414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:53.714416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:53.714507 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:53.714876 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:53.714879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:53.714881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:53.715095 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:53.715362 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:53.715935 (+ 573us) server_negotiation.cc:299] Negotiation successful
0807 06:22:53.716097 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 06:22:53.717444 25845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:53.702427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.3:36657)
0807 06:22:53.703049 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:53.703113 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:53.704864 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:53.705549 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:53.705556 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:53.705991 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:53.706522 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:53.706538 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:53.707330 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:53.707333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:53.708371 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:53.708382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:53.709030 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:53.709033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:53.709182 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:53.709982 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:53.710046 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:53.712115 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:53.713846 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:53.713852 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:53.713866 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:53.714203 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:53.714603 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:53.714606 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:53.714607 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:53.714762 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:53.715337 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:53.715342 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:53.715659 (+ 317us) client_negotiation.cc:769] Sending connection context
0807 06:22:53.715888 (+ 229us) client_negotiation.cc:240] Negotiation successful
0807 06:22:53.716302 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":139,"threads_started":1}
I20260807 06:22:53.718659 25843 heartbeater.cc:347] Connected to a master server at 127.15.254.62:41399
I20260807 06:22:53.718935 25843 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:53.719470 25843 heartbeater.cc:511] Master 127.15.254.62:41399 requested a full tablet report, sending...
I20260807 06:22:53.720643 25193 ts_manager.cc:194] Registered new tserver with Master: 043fb1b788ed4af18200209e0ac9f5c0 (127.15.254.3:43515)
I20260807 06:22:53.721241 25193 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.254.3@KRBTEST.COM'} at 127.15.254.3:36657
I20260807 06:22:53.730053 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.15.254.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.254.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.254.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.61.keytab.
Entry for principal oryx/127.15.254.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.61.keytab.
I20260807 06:22:53.760541 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.61:36519
--webserver_interface=127.15.254.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.61.keytab
--principal=oryx/127.15.254.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:41399,127.15.254.61:36519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:22:53.880602 25848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:53.880873 25848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:53.880914 25848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:53.885033 25848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:22:53.885097 25848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:53.885116 25848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:53.885155 25848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:22:53.885187 25848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:22:53.890580 25848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:41399,127.15.254.61:36519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.61.keytab
--principal=oryx/127.15.254.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.61:36519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.254.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
W20260807 06:22:53.890815 25848 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.15.254.62:41399,127.15.254.61:36519'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260807 06:22:53.891816 25848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:53.892956 25848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:53.902554 25863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:53.902547 25860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:53.902547 25859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:53.902817 25848 server_base.cc:1034] running on GCE node
I20260807 06:22:53.903591 25848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:53.904829 25848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:53.905990 25848 hybrid_clock.cc:648] HybridClock initialized: now 1786083773905957 us; error 46 us; skew 500 ppm
Aug 07 06:22:53 dist-test-slave-44d2 krb5kdc[25128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083773, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:53.908936 25848 init.cc:377] Logged in from keytab as oryx/127.15.254.61@KRBTEST.COM (short username oryx)
I20260807 06:22:53.909979 25848 webserver.cc:460] Webserver started at http://127.15.254.61:38655/ using document root <none> and password file <none>
I20260807 06:22:53.910578 25848 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:53.910673 25848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:53.910923 25848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:53.913196 25848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-1/data/instance:
uuid: "5d972ddbeae549fba9841504b5db758a"
format_stamp: "Formatted at 2026-08-07 06:22:53 on dist-test-slave-44d2"
server_key: "c37f18624c7c30e14112e1d66048c0e1"
server_key_iv: "56bc6701a159bb072fa0f43de8955581"
server_key_version: "encryptionkey@0"
I20260807 06:22:53.913810 25848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-1/wal/instance:
uuid: "5d972ddbeae549fba9841504b5db758a"
format_stamp: "Formatted at 2026-08-07 06:22:53 on dist-test-slave-44d2"
server_key: "c37f18624c7c30e14112e1d66048c0e1"
server_key_iv: "56bc6701a159bb072fa0f43de8955581"
server_key_version: "encryptionkey@0"
I20260807 06:22:53.917979 25848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:22:53.920714 25871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:53.922087 25848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 06:22:53.922257 25848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-1/wal
uuid: "5d972ddbeae549fba9841504b5db758a"
format_stamp: "Formatted at 2026-08-07 06:22:53 on dist-test-slave-44d2"
server_key: "c37f18624c7c30e14112e1d66048c0e1"
server_key_iv: "56bc6701a159bb072fa0f43de8955581"
server_key_version: "encryptionkey@0"
I20260807 06:22:53.922380 25848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:53.941895 25848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:53.945485 25848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:53.945719 25848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:53.953960 25848 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.61:36519
I20260807 06:22:53.954332 25943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.61:36519 every 8 connection(s)
I20260807 06:22:53.955102 25848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-1/data/info.pb
I20260807 06:22:53.958254 25944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:53.958694 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 25848
I20260807 06:22:53.963771 25944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:22:53 dist-test-slave-44d2 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.254.61@KRBTEST.COM
Aug 07 06:22:53 dist-test-slave-44d2 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083773, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.61@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM
I20260807 06:22:53.976213 25947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:53.960424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38814 (local address 127.15.254.61:36519)
0807 06:22:53.961061 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:53.961084 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:53.961130 (+ 46us) server_negotiation.cc:407] Connection header received
0807 06:22:53.961966 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:53.961992 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:53.962465 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:53.962868 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:53.963621 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:53.965194 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:53.966371 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:53.967115 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:53.967282 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:53.970572 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:53.970578 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:53.970589 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:53.970616 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:53.972629 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:53.973167 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:53.973175 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:53.973180 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:53.973358 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:53.973726 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:53.973729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:53.973730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:53.974258 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:53.974399 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:53.974802 (+ 403us) server_negotiation.cc:299] Negotiation successful
0807 06:22:53.975141 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":156,"threads_started":1}
I20260807 06:22:53.976631 25948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:53.965852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.0.0.1:47980)
0807 06:22:53.966470 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:53.966519 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:53.966642 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:53.966937 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:53.966944 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:53.967528 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:53.967726 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:53.967734 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:53.968623 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:53.968627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:53.969613 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:53.969627 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:53.970300 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:53.970304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:53.970421 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:53.971032 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:53.971052 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:53.972798 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:53.974802 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:53.974810 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:53.974817 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:53.975225 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:53.975494 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:53.975497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:53.975499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:53.975591 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:53.975989 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:53.975995 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:53.976111 (+ 116us) client_negotiation.cc:769] Sending connection context
0807 06:22:53.976255 (+ 144us) client_negotiation.cc:240] Negotiation successful
0807 06:22:53.976436 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":185,"threads_started":1}
I20260807 06:22:53.976694 25249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:53.965939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.15.254.62:41399)
0807 06:22:53.966104 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:53.966109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:53.966606 (+ 497us) server_negotiation.cc:407] Connection header received
0807 06:22:53.966738 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:53.966741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:53.966785 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:53.966860 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:53.967851 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:53.968415 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:53.969749 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:53.970116 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:53.970207 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:53.972939 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:53.972943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:53.972947 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:53.972983 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:53.974699 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:53.975320 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:53.975323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:53.975324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:53.975399 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:53.975681 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:53.975684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:53.975685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:53.975885 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:53.975962 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:53.976353 (+ 391us) server_negotiation.cc:299] Negotiation successful
0807 06:22:53.976483 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 06:22:53.977952 25944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:22:53 dist-test-slave-44d2 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083773, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.61@KRBTEST.COM for oryx/127.15.254.61@KRBTEST.COM
I20260807 06:22:53.986552 25948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:53.978510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:36519 (local address 127.0.0.1:38818)
0807 06:22:53.978636 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:53.978651 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:53.978730 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:53.978956 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:53.978959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:53.979100 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:53.979201 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:53.979207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:53.980302 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:53.980305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:53.980907 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:53.980917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:53.981380 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:53.981383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:53.981463 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:53.982051 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:53.982066 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:53.983354 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:53.985091 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:53.985094 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:53.985095 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:53.985325 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:53.985583 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:53.985585 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:53.985586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:53.985663 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:53.986035 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:53.986041 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:53.986155 (+ 114us) client_negotiation.cc:769] Sending connection context
0807 06:22:53.986252 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 06:22:53.986371 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260807 06:22:53.986554 25947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:53.978559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38818 (local address 127.15.254.61:36519)
0807 06:22:53.978673 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:53.978676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:53.978690 (+ 14us) server_negotiation.cc:407] Connection header received
0807 06:22:53.978806 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:53.978809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:53.978855 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:53.978935 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:53.979298 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:53.980154 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:53.981042 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:53.981269 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:53.981361 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:53.983513 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:53.983516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:53.983518 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:53.983540 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:53.984964 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:53.985417 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:53.985420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:53.985421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:53.985492 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:53.985750 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:53.985752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:53.985753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:53.985913 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:53.985988 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:53.986309 (+ 321us) server_negotiation.cc:299] Negotiation successful
0807 06:22:53.986426 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":44}
I20260807 06:22:53.989699 25944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a: Bootstrap starting.
I20260807 06:22:53.992164 25944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:53.993059 25944 log.cc:824] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:53.995150 25944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a: No bootstrap required, opened a new log
I20260807 06:22:53.998437 25944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } }
I20260807 06:22:53.998648 25944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:53.998683 25944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d972ddbeae549fba9841504b5db758a, State: Initialized, Role: FOLLOWER
I20260807 06:22:53.999148 25944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } }
I20260807 06:22:53.999637 25950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } } }
I20260807 06:22:53.999761 25950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:22:54.000435 25944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [sys.catalog]: configured and running, proceeding with master startup.
W20260807 06:22:54.005920 25971 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:22:54.006035 25971 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:22:54.008471 25848 master_runner.cc:428] Detected that this master 5d972ddbeae549fba9841504b5db758a is joining an existing cluster
I20260807 06:22:54.008576 25848 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.254.61:36519
I20260807 06:22:54.020759 25947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.011517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38820 (local address 127.15.254.61:36519)
0807 06:22:54.011779 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:54.011784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:54.012056 (+ 272us) server_negotiation.cc:407] Connection header received
0807 06:22:54.012304 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:54.012307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:54.012357 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:54.012436 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:54.013136 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.013938 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.015262 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.015580 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.015705 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.016936 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:54.016940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:54.016943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:54.016966 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:54.018777 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.019317 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.019322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.019324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.019413 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.019720 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.019726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.019728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.019925 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:54.020041 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:54.020326 (+ 285us) server_negotiation.cc:299] Negotiation successful
0807 06:22:54.020541 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":179}
I20260807 06:22:54.020826 25978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.011225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.0.0.1:47982)
0807 06:22:54.011960 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:54.011974 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:54.012111 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:54.012437 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:54.012441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:54.012776 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:54.013028 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.013039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.014140 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.014143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.014814 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.014830 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.015368 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.015371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.015520 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.016123 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:54.016145 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:54.016728 (+ 583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:54.018964 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.018968 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.018970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.019267 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.019610 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.019613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.019615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.019720 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.020222 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:54.020225 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:54.020311 (+ 86us) client_negotiation.cc:769] Sending connection context
0807 06:22:54.020415 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 06:22:54.020604 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":523,"thread_start_us":126,"threads_started":1}
I20260807 06:22:54.020792 25249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.011359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47982 (local address 127.15.254.62:41399)
0807 06:22:54.011665 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:54.011672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:54.012061 (+ 389us) server_negotiation.cc:407] Connection header received
0807 06:22:54.012210 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:54.012214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:54.012297 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:54.012402 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:54.013196 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.013995 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.014960 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.015195 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.015353 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.016895 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:54.016902 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:54.016909 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:54.016942 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:54.018817 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.019391 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.019395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.019397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.019492 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.019828 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.019832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.019834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.020069 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:54.020145 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:54.020516 (+ 371us) server_negotiation.cc:299] Negotiation successful
0807 06:22:54.020635 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209}
I20260807 06:22:54.021704 25979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.011225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:36519 (local address 127.0.0.1:38820)
0807 06:22:54.011968 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:54.011984 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:54.012163 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:54.012566 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:54.012569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:54.012806 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:54.013006 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.013014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.014085 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.014089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.015128 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.015139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.015706 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.015710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.015836 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.016281 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:54.016294 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:54.016816 (+ 522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:54.018905 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.018911 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.018916 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.019208 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.019516 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.019521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.019523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.019619 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.020041 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:54.020045 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:54.020145 (+ 100us) client_negotiation.cc:769] Sending connection context
0807 06:22:54.020281 (+ 136us) client_negotiation.cc:240] Negotiation successful
0807 06:22:54.021500 (+ 1219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":28,"spinlock_wait_cycles":768,"thread_start_us":154,"threads_started":1}
Aug 07 06:22:54 dist-test-slave-44d2 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083772, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.62@KRBTEST.COM for oryx/127.15.254.61@KRBTEST.COM
I20260807 06:22:54.036988 25980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.026168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:36519 (local address 127.0.0.1:38834)
0807 06:22:54.026601 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:54.026656 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:54.026788 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:54.027168 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:54.027174 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:54.027649 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:54.027792 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.027801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.028515 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.028519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.029510 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.029522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.030061 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.030065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.030181 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.030729 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:54.030745 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:54.032818 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:54.034707 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.034719 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.034727 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.035131 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.035509 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.035514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.035517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.035630 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.036203 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:54.036211 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:54.036364 (+ 153us) client_negotiation.cc:769] Sending connection context
0807 06:22:54.036589 (+ 225us) client_negotiation.cc:240] Negotiation successful
0807 06:22:54.036824 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":141,"threads_started":1}
I20260807 06:22:54.036983 25947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.026209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38834 (local address 127.15.254.61:36519)
0807 06:22:54.026389 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:54.026394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:54.026744 (+ 350us) server_negotiation.cc:407] Connection header received
0807 06:22:54.026967 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:54.026970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:54.027030 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:54.027117 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:54.027919 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.028391 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.029636 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.029944 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.030052 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.033038 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:54.033042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:54.033044 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:54.033068 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:54.034573 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.035276 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.035280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.035282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.035382 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.035755 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.035758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.035759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.036075 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:54.036254 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:54.036668 (+ 414us) server_negotiation.cc:299] Negotiation successful
0807 06:22:54.036804 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 06:22:54.038902 25193 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.254.61:36519: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5d972ddbeae549fba9841504b5db758a"
member_type: NON_VOTER
last_known_addr {
host: "127.15.254.61"
port: 36519
}
attrs {
promote: true
}
}
dest_uuid: "9f97442df1a34ff6a8a6215460b56d62"
cas_config_opid_index: -1
I20260807 06:22:54.039604 25193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: NON_VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: true } }
I20260807 06:22:54.042146 25986 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5d972ddbeae549fba9841504b5db758a (127.15.254.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: NON_VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: true } } }
WARNING: no policy specified for oryx/127.15.254.60@KRBTEST.COM; defaulting to no policy
I20260807 06:22:54.042807 25908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:54.045176 25986 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.254.61:36519
I20260807 06:22:54.045261 25908 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 1 FOLLOWER]: Refusing update from remote peer 9f97442df1a34ff6a8a6215460b56d62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.254.60@KRBTEST.COM" created.
I20260807 06:22:54.045928 25848 master.cc:537] Master@127.15.254.61:36519 shutting down...
I20260807 06:22:54.046074 25986 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: NON_VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 06:22:54.047644 25176 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260807 06:22:54.045436 25985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9f97442df1a34ff6a8a6215460b56d62" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: NON_VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: true } } }
I20260807 06:22:54.048842 25985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:54.052075 25848 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:22:54.052402 25848 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:54.052551 25848 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a: stopping tablet replica
I20260807 06:22:54.058415 25848 master.cc:552] Master@127.15.254.61:36519 shutdown complete.
I20260807 06:22:54.058465 25848 master_runner.cc:305] Clearing existing system tablet
I20260807 06:22:54.058755 25980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.058399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:36519 (local address 127.0.0.1:38842)
0807 06:22:54.058552 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:54.058662 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.61:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260807 06:22:54.058805 25848 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:54.058890 25848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260807 06:22:54.058923 25176 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 -> Peer 5d972ddbeae549fba9841504b5db758a (127.15.254.61:36519): Couldn't send request to peer 5d972ddbeae549fba9841504b5db758a. Status: Network error: Client connection negotiation failed: client connection to 127.15.254.61:36519: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 06:22:54.060573 25848 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 06:22:54.061645 25991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:54.061882 25848 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:54.062039 25848 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 06:22:54.062084 25848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-1/wal
uuid: "5d972ddbeae549fba9841504b5db758a"
format_stamp: "Formatted at 2026-08-07 06:22:53 on dist-test-slave-44d2"
server_key: "c37f18624c7c30e14112e1d66048c0e1"
server_key_iv: "56bc6701a159bb072fa0f43de8955581"
server_key_version: "encryptionkey@0"
I20260807 06:22:54.064169 25848 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:22:54.066823 25848 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260807 06:22:54.066942 25848 log.cc:1196] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.254.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.60.keytab.
Entry for principal oryx/127.15.254.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.60.keytab.
I20260807 06:22:54.067258 25848 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a: Deleting consensus metadata
I20260807 06:22:54.067443 25848 master_runner.cc:315] Copying system tablet from 127.15.254.62:41399
I20260807 06:22:54.068303 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.60:44007
--webserver_interface=127.15.254.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.60.keytab
--principal=oryx/127.15.254.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:41399,127.15.254.61:36519,127.15.254.60:44007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:22:54.068783 25848 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.254.62:41399
I20260807 06:22:54.079681 25249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.069375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47992 (local address 127.15.254.62:41399)
0807 06:22:54.069499 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:54.069504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:54.069737 (+ 233us) server_negotiation.cc:407] Connection header received
0807 06:22:54.069899 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:54.069902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:54.069959 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:54.070070 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:54.070785 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.071628 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.073037 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.073402 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.073521 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.075688 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:54.075693 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:54.075697 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:54.075728 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:54.077593 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.078169 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.078173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.078175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.078274 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.078640 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.078643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.078645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.078897 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:54.079003 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:54.079362 (+ 359us) server_negotiation.cc:299] Negotiation successful
0807 06:22:54.079507 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 06:22:54.079809 26001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.069320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.0.0.1:47992)
0807 06:22:54.069636 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:54.069649 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:54.069804 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:54.070228 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:54.070234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:54.070477 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:54.070635 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.070645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.071771 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.071777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.072886 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.072903 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.073560 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.073564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.073688 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.074924 (+ 1236us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:54.074947 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:54.075496 (+ 549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:54.077727 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.077731 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.077734 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.078038 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.078386 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.078392 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.078396 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.078510 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.079015 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:54.079021 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:54.079160 (+ 139us) client_negotiation.cc:769] Sending connection context
0807 06:22:54.079288 (+ 128us) client_negotiation.cc:240] Negotiation successful
0807 06:22:54.079497 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":84,"threads_started":1}
I20260807 06:22:54.080736 25221 tablet_copy_service.cc:140] P 9f97442df1a34ff6a8a6215460b56d62: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5d972ddbeae549fba9841504b5db758a ({username='oryx', principal='oryx/127.15.254.61@KRBTEST.COM'} at 127.0.0.1:47992)
I20260807 06:22:54.080971 25221 tablet_copy_service.cc:161] P 9f97442df1a34ff6a8a6215460b56d62: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5d972ddbeae549fba9841504b5db758a at {username='oryx', principal='oryx/127.15.254.61@KRBTEST.COM'} at 127.0.0.1:47992: session id = 5d972ddbeae549fba9841504b5db758a-00000000000000000000000000000000
I20260807 06:22:54.083431 25221 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 06:22:54.085384 25848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260807 06:22:54.087852 25848 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a: tablet copy: Starting download of 0 data blocks...
I20260807 06:22:54.088068 25848 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a: tablet copy: Starting download of 1 WAL segments...
I20260807 06:22:54.090041 25848 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 06:22:54.091869 25221 tablet_copy_service.cc:342] P 9f97442df1a34ff6a8a6215460b56d62: Request end of tablet copy session 5d972ddbeae549fba9841504b5db758a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.254.61@KRBTEST.COM'} at 127.0.0.1:47992
I20260807 06:22:54.092080 25221 tablet_copy_service.cc:433] P 9f97442df1a34ff6a8a6215460b56d62: ending tablet copy session 5d972ddbeae549fba9841504b5db758a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5d972ddbeae549fba9841504b5db758a
I20260807 06:22:54.099797 25848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:54.101609 26004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:54.101692 26005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:54.101843 26007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:54.109278 25848 server_base.cc:1034] running on GCE node
I20260807 06:22:54.109505 25848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:54.109942 25848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:54.111110 25848 hybrid_clock.cc:648] HybridClock initialized: now 1786083774111087 us; error 42 us; skew 500 ppm
Aug 07 06:22:54 dist-test-slave-44d2 krb5kdc[25128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083774, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:54.112892 25848 init.cc:377] Logged in from keytab as oryx/127.15.254.61@KRBTEST.COM (short username oryx)
I20260807 06:22:54.113451 25848 webserver.cc:460] Webserver started at http://127.15.254.61:34811/ using document root <none> and password file <none>
I20260807 06:22:54.113658 25848 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:54.113705 25848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:54.115192 25848 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260807 06:22:54.116395 26015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:54.116770 25848 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 06:22:54.116861 25848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-1/wal
uuid: "5d972ddbeae549fba9841504b5db758a"
format_stamp: "Formatted at 2026-08-07 06:22:53 on dist-test-slave-44d2"
server_key: "c37f18624c7c30e14112e1d66048c0e1"
server_key_iv: "56bc6701a159bb072fa0f43de8955581"
server_key_version: "encryptionkey@0"
I20260807 06:22:54.116950 25848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:54.130309 25848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:54.132637 25848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:54.138351 25848 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.61:36519
I20260807 06:22:54.138482 26087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.61:36519 every 8 connection(s)
I20260807 06:22:54.138996 25848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-1/data/info.pb
I20260807 06:22:54.141002 26090 sys_catalog.cc:263] Verifying existing consensus state
I20260807 06:22:54.142086 26090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a: Bootstrap starting.
I20260807 06:22:54.152974 26090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 06:22:54.153403 26090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a: Bootstrap complete.
I20260807 06:22:54.154134 26090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: NON_VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: true } }
I20260807 06:22:54.154301 26090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5d972ddbeae549fba9841504b5db758a, State: Initialized, Role: LEARNER
I20260807 06:22:54.154428 26090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: NON_VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: true } }
I20260807 06:22:54.154969 26090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:22:54.154964 26096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: NON_VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: true } } }
I20260807 06:22:54.155241 26096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [sys.catalog]: This master's current role is: LEARNER
I20260807 06:22:54.159049 26109 catalog_manager.cc:1259] Loaded cluster ID: b67ecb0da3614575836bf6cf3f400654
I20260807 06:22:54.159125 26109 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a: loading cluster ID for follower catalog manager: success
I20260807 06:22:54.161378 26109 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a: acquiring CA information for follower catalog manager: success
I20260807 06:22:54.162918 26109 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260807 06:22:54.211814 26002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:54.212122 26002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:54.212225 26002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:54.216277 26002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:22:54.216370 26002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:54.216411 26002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:54.216464 26002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:22:54.216521 26002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:22:54.222437 26002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:41399,127.15.254.61:36519,127.15.254.60:44007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.60.keytab
--principal=oryx/127.15.254.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.60:44007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.254.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:54.223595 26002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:54.224709 26002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:54.233194 26117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:54.233191 26119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:54.233290 26114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:54.234468 26002 server_base.cc:1034] running on GCE node
I20260807 06:22:54.235054 26002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:54.236225 26002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:54.237423 26002 hybrid_clock.cc:648] HybridClock initialized: now 1786083774237367 us; error 52 us; skew 500 ppm
Aug 07 06:22:54 dist-test-slave-44d2 krb5kdc[25128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083774, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:54.240639 26002 init.cc:377] Logged in from keytab as oryx/127.15.254.60@KRBTEST.COM (short username oryx)
I20260807 06:22:54.241542 26002 webserver.cc:460] Webserver started at http://127.15.254.60:38663/ using document root <none> and password file <none>
I20260807 06:22:54.242136 26002 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:54.242203 26002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:54.242475 26002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:22:54.244746 26002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-2/data/instance:
uuid: "8e73d6f7d63a48379cbc6aa648137096"
format_stamp: "Formatted at 2026-08-07 06:22:54 on dist-test-slave-44d2"
server_key: "1f6b9439d28ae96cca032adb6a892637"
server_key_iv: "cba3b92300fb0c21698e78e152eef799"
server_key_version: "encryptionkey@0"
I20260807 06:22:54.245314 26002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-2/wal/instance:
uuid: "8e73d6f7d63a48379cbc6aa648137096"
format_stamp: "Formatted at 2026-08-07 06:22:54 on dist-test-slave-44d2"
server_key: "1f6b9439d28ae96cca032adb6a892637"
server_key_iv: "cba3b92300fb0c21698e78e152eef799"
server_key_version: "encryptionkey@0"
I20260807 06:22:54.249105 26002 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:22:54.251554 26129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:54.252936 26002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 06:22:54.253096 26002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-2/wal
uuid: "8e73d6f7d63a48379cbc6aa648137096"
format_stamp: "Formatted at 2026-08-07 06:22:54 on dist-test-slave-44d2"
server_key: "1f6b9439d28ae96cca032adb6a892637"
server_key_iv: "cba3b92300fb0c21698e78e152eef799"
server_key_version: "encryptionkey@0"
I20260807 06:22:54.253217 26002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:54.255270 25457 heartbeater.cc:503] Master 127.15.254.62:41399 was elected leader, sending a full tablet report...
I20260807 06:22:54.278491 26002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:54.281854 26002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:54.282037 26002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:54.289955 26002 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.60:44007
I20260807 06:22:54.290254 26204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.60:44007 every 8 connection(s)
I20260807 06:22:54.291100 26002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-2/data/info.pb
I20260807 06:22:54.294251 26206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:54.297317 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 26002
I20260807 06:22:54.298472 26206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:22:54 dist-test-slave-44d2 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083774, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.60@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM
Aug 07 06:22:54 dist-test-slave-44d2 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.254.60@KRBTEST.COM
I20260807 06:22:54.311419 25249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.300405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47996 (local address 127.15.254.62:41399)
0807 06:22:54.300558 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:54.300572 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:54.300747 (+ 175us) server_negotiation.cc:407] Connection header received
0807 06:22:54.301092 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:54.301095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:54.301161 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:54.301266 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:54.302101 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.302651 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.303801 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.304034 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.304204 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.306997 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:54.307001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:54.307003 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:54.307035 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:54.308916 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.309634 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.309638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.309640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.309729 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.310142 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.310146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.310148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.310336 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:54.310420 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:54.311114 (+ 694us) server_negotiation.cc:299] Negotiation successful
0807 06:22:54.311238 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 06:22:54.312602 26212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.300190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.0.0.1:47996)
0807 06:22:54.300613 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:54.300655 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:54.300907 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:54.301323 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:54.301329 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:54.301688 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:54.301942 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.301956 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.302814 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.302818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.303677 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.303687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.304247 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.304251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.304389 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.305161 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:54.305182 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:54.306850 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:54.309054 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.309063 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.309073 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.309509 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.309831 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.309835 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.309837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.310025 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.310478 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:54.310483 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:54.310796 (+ 313us) client_negotiation.cc:769] Sending connection context
0807 06:22:54.311033 (+ 237us) client_negotiation.cc:240] Negotiation successful
0807 06:22:54.311291 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":1,"thread_start_us":75,"threads_started":1}
I20260807 06:22:54.312606 26209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.299134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43532 (local address 127.15.254.60:44007)
0807 06:22:54.299566 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:54.299580 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:54.299615 (+ 35us) server_negotiation.cc:407] Connection header received
0807 06:22:54.300532 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:54.300554 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:54.300930 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:54.301159 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:54.301983 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.302906 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.303739 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.304234 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.304391 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.307532 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:54.307537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:54.307547 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:54.307568 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:54.309590 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.310476 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.310481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.310482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.310552 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.311010 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.311014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.311017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.311311 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:54.311538 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:54.311837 (+ 299us) server_negotiation.cc:299] Negotiation successful
0807 06:22:54.311950 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":286,"thread_start_us":147,"threads_started":1}
I20260807 06:22:54.314575 26206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:22:54 dist-test-slave-44d2 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083774, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.60@KRBTEST.COM for oryx/127.15.254.61@KRBTEST.COM
I20260807 06:22:54.323451 26212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.315177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:36519 (local address 127.0.0.1:38848)
0807 06:22:54.315311 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:54.315329 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:54.315409 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:54.315922 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:54.315924 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:54.316083 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:54.316203 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.316208 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.317252 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.317255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.317842 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.317851 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.318332 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.318336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.318418 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.318942 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:54.318954 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:54.320273 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:54.322042 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.322045 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.322047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.322276 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.322546 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.322548 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.322549 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.322629 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.323040 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:54.323042 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:54.323139 (+ 97us) client_negotiation.cc:769] Sending connection context
0807 06:22:54.323219 (+ 80us) client_negotiation.cc:240] Negotiation successful
0807 06:22:54.323325 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260807 06:22:54.323535 26213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.315410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38848 (local address 127.15.254.61:36519)
0807 06:22:54.315706 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:54.315710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:54.315722 (+ 12us) server_negotiation.cc:407] Connection header received
0807 06:22:54.315762 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:54.315764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:54.315805 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:54.315883 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:54.316317 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.317114 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.317956 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.318203 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.318287 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.320412 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:54.320415 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:54.320417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:54.320441 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:54.321939 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.322373 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.322376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.322377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.322459 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.322709 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.322712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.322713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.322953 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:54.323036 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:54.323293 (+ 257us) server_negotiation.cc:299] Negotiation successful
0807 06:22:54.323404 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":123,"threads_started":1}
I20260807 06:22:54.324601 26206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.60" port: 44007 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:22:54 dist-test-slave-44d2 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083774, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.60@KRBTEST.COM for oryx/127.15.254.60@KRBTEST.COM
I20260807 06:22:54.332738 26212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.325168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:44007 (local address 127.0.0.1:43544)
0807 06:22:54.325299 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:54.325315 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:54.325387 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:54.325593 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:54.325595 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:54.325724 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:54.325794 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.325798 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.326634 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.326637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.327134 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.327141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.328210 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.328213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.328270 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.328817 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:54.328829 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:54.330005 (+ 1176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:54.331484 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.331486 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.331488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.331672 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.331907 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.331909 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.331911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.331982 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.332297 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:54.332299 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:54.332371 (+ 72us) client_negotiation.cc:769] Sending connection context
0807 06:22:54.332451 (+ 80us) client_negotiation.cc:240] Negotiation successful
0807 06:22:54.332623 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260807 06:22:54.332836 26209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.325257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43544 (local address 127.15.254.60:44007)
0807 06:22:54.325357 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:54.325360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:54.325371 (+ 11us) server_negotiation.cc:407] Connection header received
0807 06:22:54.325460 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:54.325462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:54.325497 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:54.325561 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:54.325996 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.326526 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.327242 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.328106 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.328196 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.330128 (+ 1932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:54.330131 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:54.330133 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:54.330153 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:54.331379 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.331756 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.331758 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.331759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.331827 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.332060 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.332062 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.332063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.332216 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:54.332289 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:54.332601 (+ 312us) server_negotiation.cc:299] Negotiation successful
0807 06:22:54.332689 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260807 06:22:54.335807 26206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096: Bootstrap starting.
I20260807 06:22:54.338312 26206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:54.339023 26206 log.cc:824] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:54.341024 26206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096: No bootstrap required, opened a new log
I20260807 06:22:54.344184 26206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 44007 } }
I20260807 06:22:54.344393 26206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:54.344426 26206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e73d6f7d63a48379cbc6aa648137096, State: Initialized, Role: FOLLOWER
I20260807 06:22:54.344902 26206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 44007 } }
I20260807 06:22:54.345358 26215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 44007 } } }
I20260807 06:22:54.345482 26215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:22:54.346156 26206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [sys.catalog]: configured and running, proceeding with master startup.
W20260807 06:22:54.351081 26230 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:22:54.351176 26230 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:22:54.355144 26002 master_runner.cc:428] Detected that this master 8e73d6f7d63a48379cbc6aa648137096 is joining an existing cluster
I20260807 06:22:54.355227 26002 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.254.60:44007
I20260807 06:22:54.366176 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 25273
I20260807 06:22:54.390627 26209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.358924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43546 (local address 127.15.254.60:44007)
0807 06:22:54.359072 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:54.359078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:54.359567 (+ 489us) server_negotiation.cc:407] Connection header received
0807 06:22:54.359736 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:54.359740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:54.359779 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:54.359833 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:54.360719 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.361443 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.362574 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.362925 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.363022 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.364589 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:54.364594 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:54.364599 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:54.364634 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:54.387303 (+ 22669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.387986 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.387991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.387993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.388083 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.388391 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.388394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.388396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.388666 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:54.388753 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:54.390179 (+ 1426us) server_negotiation.cc:299] Negotiation successful
0807 06:22:54.390335 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":59}
I20260807 06:22:54.392692 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.359123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:44007 (local address 127.0.0.1:43546)
0807 06:22:54.359477 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:54.359492 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:54.359630 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:54.359966 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:54.359968 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:54.360348 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:54.360578 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.360590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.361589 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.361592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.362434 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.362447 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.363054 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.363058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.363213 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.363852 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:54.363870 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:54.364414 (+ 544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:54.387491 (+ 23077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.387496 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.387498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.387848 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.388195 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.388198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.388201 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.388283 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.388852 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:54.388855 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:54.389542 (+ 687us) client_negotiation.cc:769] Sending connection context
0807 06:22:54.392092 (+ 2550us) client_negotiation.cc:240] Negotiation successful
0807 06:22:54.392267 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":49,"thread_start_us":114,"threads_started":1}
I20260807 06:22:54.393545 25249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.358095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48008 (local address 127.15.254.62:41399)
0807 06:22:54.358257 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:54.358264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:54.359542 (+ 1278us) server_negotiation.cc:407] Connection header received
0807 06:22:54.359735 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:54.359740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:54.359802 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:54.359859 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:54.360646 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.361552 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.363563 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.363898 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.363961 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.388551 (+ 24590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:54.388558 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:54.388584 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:54.388626 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:54.391066 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.391640 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.391645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.391647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.391745 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.392084 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.392088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.392090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.392343 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:54.392424 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:54.393206 (+ 782us) server_negotiation.cc:299] Negotiation successful
0807 06:22:54.393369 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 06:22:54.394163 26238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.358375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.0.0.1:48008)
0807 06:22:54.359447 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:54.359462 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:54.359612 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:54.359998 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:54.360001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:54.360314 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:54.360508 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.360517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.361708 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.361713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.363395 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.363405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.364102 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.364104 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.364173 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.364723 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:54.364732 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:54.387769 (+ 23037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:54.391221 (+ 3452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.391225 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.391227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.391507 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.391873 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.391876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.391879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.391974 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.392530 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:54.392533 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:54.392991 (+ 458us) client_negotiation.cc:769] Sending connection context
0807 06:22:54.393774 (+ 783us) client_negotiation.cc:240] Negotiation successful
0807 06:22:54.393955 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"mutex_wait_us":23,"spinlock_wait_cycles":896,"thread_start_us":212,"threads_started":1}
I20260807 06:22:54.394439 26213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.358624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38864 (local address 127.15.254.61:36519)
0807 06:22:54.358761 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:54.358767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:54.358783 (+ 16us) server_negotiation.cc:407] Connection header received
0807 06:22:54.358975 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:54.358982 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:54.359036 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:54.359094 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:54.359849 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.360679 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.361979 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.362369 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.362497 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.389047 (+ 26550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:54.389054 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:54.389061 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:54.389098 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:54.391585 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.392941 (+ 1356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.392945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.392947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.393053 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.393411 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.393415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.393416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.393663 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:54.393751 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:54.394122 (+ 371us) server_negotiation.cc:299] Negotiation successful
0807 06:22:54.394273 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 06:22:54.395010 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.358088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:36519 (local address 127.0.0.1:38864)
0807 06:22:54.358710 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:54.358726 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:54.358831 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:54.359228 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:54.359230 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:54.359526 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:54.359713 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.359720 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.360811 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.360814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.361852 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.361860 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.362762 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.362768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.363182 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.386377 (+ 23195us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:54.386405 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:54.388300 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:54.391723 (+ 3423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.391729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.391734 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.392005 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.393183 (+ 1178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.393186 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.393188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.393282 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.393826 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:54.393829 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:54.393918 (+ 89us) client_negotiation.cc:769] Sending connection context
0807 06:22:54.394679 (+ 761us) client_negotiation.cc:240] Negotiation successful
0807 06:22:54.394845 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":67,"thread_start_us":157,"threads_started":1}
I20260807 06:22:54.397471 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:42279
--local_ip_for_outbound_sockets=127.15.254.1
--tserver_master_addrs=127.15.254.62:41399
--webserver_port=37981
--webserver_interface=127.15.254.1
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.1.keytab
--principal=oryx/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.254.62:41399,127.15.254.61:36519
--tserver_master_addrs=127.15.254.62:41399,127.15.254.61:36519,127.15.254.60:44007 with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 06:22:54 dist-test-slave-44d2 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083772, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.62@KRBTEST.COM for oryx/127.15.254.60@KRBTEST.COM
I20260807 06:22:54.407694 26209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.398870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43548 (local address 127.15.254.60:44007)
0807 06:22:54.399554 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:54.399560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:54.399586 (+ 26us) server_negotiation.cc:407] Connection header received
0807 06:22:54.399667 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:54.399671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:54.399734 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:54.399826 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:54.400352 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.401165 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.401965 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.402197 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.402306 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.404411 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:54.404414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:54.404416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:54.404438 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:54.406003 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.406469 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.406473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.406475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.406556 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.406854 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.406857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.406859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.407032 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:54.407150 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:54.407433 (+ 283us) server_negotiation.cc:299] Negotiation successful
0807 06:22:54.407541 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586}
I20260807 06:22:54.407696 25980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.398786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:44007 (local address 127.0.0.1:43548)
0807 06:22:54.398974 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:54.398989 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:54.399072 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:54.399876 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:54.399880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:54.400089 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:54.400212 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.400221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.401279 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.401283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.401833 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.401841 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.402319 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.402323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.402430 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.402924 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:54.402937 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:54.404260 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:54.406146 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.406149 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.406151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.406362 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.406655 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.406659 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.406661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.406752 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.407144 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:54.407147 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:54.407255 (+ 108us) client_negotiation.cc:769] Sending connection context
0807 06:22:54.407365 (+ 110us) client_negotiation.cc:240] Negotiation successful
0807 06:22:54.407518 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260807 06:22:54.409152 25193 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.254.60:44007: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096"
member_type: NON_VOTER
last_known_addr {
host: "127.15.254.60"
port: 44007
}
attrs {
promote: true
}
}
dest_uuid: "9f97442df1a34ff6a8a6215460b56d62"
cas_config_opid_index: 5
I20260807 06:22:54.409829 25193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: NON_VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: true } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: NON_VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: true } }
I20260807 06:22:54.410996 25985 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8e73d6f7d63a48379cbc6aa648137096 (127.15.254.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: NON_VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: true } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: NON_VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: true } } }
I20260807 06:22:54.411646 26166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:54.413587 25985 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.254.60:44007
I20260807 06:22:54.413779 25985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9f97442df1a34ff6a8a6215460b56d62" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: NON_VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: true } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: NON_VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: true } } }
I20260807 06:22:54.413841 26166 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 1 FOLLOWER]: Refusing update from remote peer 9f97442df1a34ff6a8a6215460b56d62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260807 06:22:54.413870 25985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:54.414304 26002 master.cc:537] Master@127.15.254.60:44007 shutting down...
I20260807 06:22:54.415015 25986 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: NON_VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 06:22:54.417629 25179 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 -> Peer 8e73d6f7d63a48379cbc6aa648137096 (127.15.254.60:44007): Couldn't send request to peer 8e73d6f7d63a48379cbc6aa648137096. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260807 06:22:54.420544 26002 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:22:54.420866 26002 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:54.420974 26002 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096: stopping tablet replica
I20260807 06:22:54.422024 25980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.412861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:36519 (local address 127.0.0.1:38866)
0807 06:22:54.416946 (+ 4085us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:54.416963 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:54.417072 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:54.417699 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:54.417702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:54.417723 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:54.417825 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.417833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.419040 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.419046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.420452 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.420465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.421618 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.421621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.421688 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.421721 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 06:22:54.421820 (+ 99us) client_negotiation.cc:240] Negotiation successful
0807 06:22:54.421893 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3985}
I20260807 06:22:54.422024 26213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.413118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38866 (local address 127.15.254.61:36519)
0807 06:22:54.413239 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:54.413245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:54.417204 (+ 3959us) server_negotiation.cc:407] Connection header received
0807 06:22:54.417288 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:54.417291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:54.417373 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:54.417450 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:54.417973 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.418763 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.420634 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.421248 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.421331 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.421494 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:54.421838 (+ 344us) server_negotiation.cc:299] Negotiation successful
0807 06:22:54.421911 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 06:22:54.422843 26045 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 1 LEARNER]: Refusing update from remote peer 9f97442df1a34ff6a8a6215460b56d62: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260807 06:22:54.423383 25986 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: NON_VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 06:22:54.425400 26045 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8e73d6f7d63a48379cbc6aa648137096 (127.15.254.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: NON_VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: true } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: NON_VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: true } } }
I20260807 06:22:54.427500 26096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f97442df1a34ff6a8a6215460b56d62. Latest consensus state: current_term: 1 leader_uuid: "9f97442df1a34ff6a8a6215460b56d62" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: NON_VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: true } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: NON_VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: true } } }
I20260807 06:22:54.427635 26096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [sys.catalog]: This master's current role is: LEARNER
I20260807 06:22:54.427750 26096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f97442df1a34ff6a8a6215460b56d62" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: NON_VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: true } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: NON_VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: true } } }
I20260807 06:22:54.427839 26096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [sys.catalog]: This master's current role is: LEARNER
I20260807 06:22:54.429221 25986 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62: attempting to promote NON_VOTER 5d972ddbeae549fba9841504b5db758a to VOTER
I20260807 06:22:54.429708 25986 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: NON_VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: true } }
I20260807 06:22:54.431238 26045 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 1 LEARNER]: Refusing update from remote peer 9f97442df1a34ff6a8a6215460b56d62: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260807 06:22:54.431676 26243 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 06:22:54.431680 25980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.431052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:44007 (local address 127.0.0.1:43550)
0807 06:22:54.431524 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:54.431598 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392}
I20260807 06:22:54.433600 25980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.433326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:44007 (local address 127.0.0.1:43560)
0807 06:22:54.433471 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:54.433519 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260807 06:22:54.433821 25179 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 -> Peer 8e73d6f7d63a48379cbc6aa648137096 (127.15.254.60:44007): Couldn't send request to peer 8e73d6f7d63a48379cbc6aa648137096. Status: Network error: Client connection negotiation failed: client connection to 127.15.254.60:44007: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 06:22:54.434156 25985 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5d972ddbeae549fba9841504b5db758a (127.15.254.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: NON_VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: true } } }
I20260807 06:22:54.435429 25985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9f97442df1a34ff6a8a6215460b56d62" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: NON_VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: true } } }
I20260807 06:22:54.435549 25985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:54.435685 26243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f97442df1a34ff6a8a6215460b56d62" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: NON_VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: true } } }
I20260807 06:22:54.435838 26243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:54.436241 26045 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5d972ddbeae549fba9841504b5db758a (127.15.254.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: NON_VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: true } } }
I20260807 06:22:54.438002 26242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f97442df1a34ff6a8a6215460b56d62" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: NON_VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: true } } }
I20260807 06:22:54.438144 26242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:22:54.444036 26002 master.cc:552] Master@127.15.254.60:44007 shutdown complete.
I20260807 06:22:54.444113 26002 master_runner.cc:305] Clearing existing system tablet
I20260807 06:22:54.444361 26002 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:54.444437 26002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:54.445699 26002 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 06:22:54.446694 26247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:54.446991 26002 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:54.447149 26002 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 06:22:54.447194 26002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-2/wal
uuid: "8e73d6f7d63a48379cbc6aa648137096"
format_stamp: "Formatted at 2026-08-07 06:22:54 on dist-test-slave-44d2"
server_key: "1f6b9439d28ae96cca032adb6a892637"
server_key_iv: "cba3b92300fb0c21698e78e152eef799"
server_key_version: "encryptionkey@0"
I20260807 06:22:54.449154 26002 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:22:54.451076 26002 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260807 06:22:54.451191 26002 log.cc:1196] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260807 06:22:54.451462 26002 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096: Deleting consensus metadata
I20260807 06:22:54.451651 26002 master_runner.cc:315] Copying system tablet from 127.15.254.62:41399
I20260807 06:22:54.452950 26002 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096: tablet copy: Beginning tablet copy session from remote peer at address 127.15.254.62:41399
I20260807 06:22:54.462088 25249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.453622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48016 (local address 127.15.254.62:41399)
0807 06:22:54.453767 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:54.453774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:54.454063 (+ 289us) server_negotiation.cc:407] Connection header received
0807 06:22:54.454222 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:54.454225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:54.454282 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:54.454371 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:54.454919 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.455733 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.456984 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.457228 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.457339 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.458592 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:54.458596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:54.458599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:54.458632 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:54.460205 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.460765 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.460769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.460771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.460863 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.461170 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.461175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.461176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.461379 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:54.461496 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:54.461787 (+ 291us) server_negotiation.cc:299] Negotiation successful
0807 06:22:54.461906 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 06:22:54.462114 26256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.453534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.0.0.1:48016)
0807 06:22:54.453958 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:54.453970 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:54.454120 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:54.454391 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:54.454394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:54.454625 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:54.454735 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.454741 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.455844 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.455849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.456862 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.456873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.457358 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.457362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.457477 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.458056 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:54.458081 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:54.458430 (+ 349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:54.460352 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.460356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.460358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.460657 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.460964 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.460967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.460969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.461072 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.461485 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:54.461488 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:54.461606 (+ 118us) client_negotiation.cc:769] Sending connection context
0807 06:22:54.461720 (+ 114us) client_negotiation.cc:240] Negotiation successful
0807 06:22:54.461894 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":109,"threads_started":1}
I20260807 06:22:54.462894 25221 tablet_copy_service.cc:140] P 9f97442df1a34ff6a8a6215460b56d62: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8e73d6f7d63a48379cbc6aa648137096 ({username='oryx', principal='oryx/127.15.254.60@KRBTEST.COM'} at 127.0.0.1:48016)
I20260807 06:22:54.463035 25221 tablet_copy_service.cc:161] P 9f97442df1a34ff6a8a6215460b56d62: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8e73d6f7d63a48379cbc6aa648137096 at {username='oryx', principal='oryx/127.15.254.60@KRBTEST.COM'} at 127.0.0.1:48016: session id = 8e73d6f7d63a48379cbc6aa648137096-00000000000000000000000000000000
I20260807 06:22:54.463989 25221 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 06:22:54.465334 26002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260807 06:22:54.468226 26002 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096: tablet copy: Starting download of 0 data blocks...
I20260807 06:22:54.468437 26002 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096: tablet copy: Starting download of 1 WAL segments...
I20260807 06:22:54.471766 26002 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 06:22:54.473588 25221 tablet_copy_service.cc:342] P 9f97442df1a34ff6a8a6215460b56d62: Request end of tablet copy session 8e73d6f7d63a48379cbc6aa648137096-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.254.60@KRBTEST.COM'} at 127.0.0.1:48016
I20260807 06:22:54.473701 25221 tablet_copy_service.cc:433] P 9f97442df1a34ff6a8a6215460b56d62: ending tablet copy session 8e73d6f7d63a48379cbc6aa648137096-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8e73d6f7d63a48379cbc6aa648137096
I20260807 06:22:54.481333 26002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:54.483305 26258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:54.483631 26259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:54.483732 26002 server_base.cc:1034] running on GCE node
W20260807 06:22:54.484004 26261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:54.484274 26002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:54.484680 26002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:54.485818 26002 hybrid_clock.cc:648] HybridClock initialized: now 1786083774485806 us; error 33 us; skew 500 ppm
Aug 07 06:22:54 dist-test-slave-44d2 krb5kdc[25128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083774, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:54.487370 26002 init.cc:377] Logged in from keytab as oryx/127.15.254.60@KRBTEST.COM (short username oryx)
I20260807 06:22:54.487859 26002 webserver.cc:460] Webserver started at http://127.15.254.60:42501/ using document root <none> and password file <none>
I20260807 06:22:54.488090 26002 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:54.488147 26002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:54.489387 26002 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 06:22:54.490356 26269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:54.490604 26002 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 06:22:54.490701 26002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-2/wal
uuid: "8e73d6f7d63a48379cbc6aa648137096"
format_stamp: "Formatted at 2026-08-07 06:22:54 on dist-test-slave-44d2"
server_key: "1f6b9439d28ae96cca032adb6a892637"
server_key_iv: "cba3b92300fb0c21698e78e152eef799"
server_key_version: "encryptionkey@0"
I20260807 06:22:54.490813 26002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:54.492347 25646 heartbeater.cc:503] Master 127.15.254.62:41399 was elected leader, sending a full tablet report...
I20260807 06:22:54.527530 26002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:54.529848 26002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:54.535405 26002 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.60:44007
I20260807 06:22:54.535580 26346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.60:44007 every 8 connection(s)
I20260807 06:22:54.536026 26002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/master-2/data/info.pb
I20260807 06:22:54.537921 26347 sys_catalog.cc:263] Verifying existing consensus state
I20260807 06:22:54.538826 26347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096: Bootstrap starting.
I20260807 06:22:54.548111 26347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 06:22:54.548436 26347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096: Bootstrap complete.
I20260807 06:22:54.548992 26347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: NON_VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: true } }
I20260807 06:22:54.549150 26347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8e73d6f7d63a48379cbc6aa648137096, State: Initialized, Role: LEARNER
I20260807 06:22:54.549271 26347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: NON_VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: true } }
I20260807 06:22:54.549655 26352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: NON_VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: true } } }
I20260807 06:22:54.549825 26347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:22:54.549839 26352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [sys.catalog]: This master's current role is: LEARNER
I20260807 06:22:54.553877 26364 catalog_manager.cc:1259] Loaded cluster ID: b67ecb0da3614575836bf6cf3f400654
I20260807 06:22:54.553972 26364 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096: loading cluster ID for follower catalog manager: success
I20260807 06:22:54.556036 26364 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096: acquiring CA information for follower catalog manager: success
I20260807 06:22:54.557417 26364 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260807 06:22:54.569295 26240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:54.569686 26240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:54.569759 26240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:54.575925 26240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:54.576017 26240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:54.576136 26240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:22:54.584632 26240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.1.keytab
--principal=oryx/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:42279
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=37981
--tserver_master_addrs=127.15.254.62:41399,127.15.254.61:36519,127.15.254.60:44007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:54.586282 26240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:54.587615 26240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:54.599614 26376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:54.599709 26369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:54.599732 26370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:54.599819 26240 server_base.cc:1034] running on GCE node
I20260807 06:22:54.600615 26240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:54.601336 26240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:54.602553 26240 hybrid_clock.cc:648] HybridClock initialized: now 1786083774602534 us; error 31 us; skew 500 ppm
Aug 07 06:22:54 dist-test-slave-44d2 krb5kdc[25128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083774, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:54.605576 26240 init.cc:377] Logged in from keytab as oryx/127.15.254.1@KRBTEST.COM (short username oryx)
I20260807 06:22:54.606477 26240 webserver.cc:460] Webserver started at http://127.15.254.1:37981/ using document root <none> and password file <none>
I20260807 06:22:54.607064 26240 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:54.607178 26240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:54.611397 26240 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:22:54.613600 26385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:54.614882 26240 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 06:22:54.615043 26240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "dc9fbde76a594443bf8b835f3b566d63"
format_stamp: "Formatted at 2026-08-07 06:22:53 on dist-test-slave-44d2"
server_key: "133d5441ba54cbd2b3c587b8effad3c1"
server_key_iv: "5e52ed9d1d9efc88f5ed35d09690b7ae"
server_key_version: "encryptionkey@0"
I20260807 06:22:54.615648 26240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:54.627508 26240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:54.630612 26240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:54.630827 26240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:54.631458 26240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:54.632395 26240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:54.632468 26240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:54.632565 26240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:54.632627 26240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:54.645116 26240 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:42279
I20260807 06:22:54.645171 26547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:42279 every 8 connection(s)
I20260807 06:22:54.646323 26240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:22:54.651311 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 26240
I20260807 06:22:54.651549 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 25464
Aug 07 06:22:54 dist-test-slave-44d2 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083774, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM
I20260807 06:22:54.669143 25249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.652583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:42581 (local address 127.15.254.62:41399)
0807 06:22:54.652728 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:54.652731 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:54.654766 (+ 2035us) server_negotiation.cc:407] Connection header received
0807 06:22:54.655094 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:54.655100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:54.655178 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:54.655284 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:54.656551 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.657338 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.658701 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.659045 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.659165 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.663547 (+ 4382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:54.663554 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:54.663559 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:54.663594 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:54.666202 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.666923 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.666931 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.666936 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.667101 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.667662 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.667669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.667674 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.668027 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:54.668179 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:54.668780 (+ 601us) server_negotiation.cc:299] Negotiation successful
0807 06:22:54.668929 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Aug 07 06:22:54 dist-test-slave-44d2 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083774, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.61@KRBTEST.COM
I20260807 06:22:54.671011 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.650768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.1:42581)
0807 06:22:54.654647 (+ 3879us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:54.654667 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:54.654868 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:54.655596 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:54.655602 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:54.656028 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:54.656366 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.656383 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.657600 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.657603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.658551 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.658561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.659262 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.659273 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.659619 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.660647 (+ 1028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:54.660687 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:54.663282 (+ 2595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:54.666395 (+ 3113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.666404 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.666422 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.666780 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.667299 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.667304 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.667308 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.667524 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.668164 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:54.668174 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:54.668500 (+ 326us) client_negotiation.cc:769] Sending connection context
0807 06:22:54.669416 (+ 916us) client_negotiation.cc:240] Negotiation successful
0807 06:22:54.669806 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3778,"thread_start_us":135,"threads_started":1}
I20260807 06:22:54.675676 26557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.649554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:36519 (local address 127.15.254.1:50859)
0807 06:22:54.650747 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:54.650761 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:54.654051 (+ 3290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:54.654900 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:54.654908 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:54.655473 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:54.656200 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.656229 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.658712 (+ 2483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.658715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.659518 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.659528 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.660007 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.660010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.660114 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.668997 (+ 8883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:54.669011 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:54.670905 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:54.673453 (+ 2548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.673457 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.673459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.673716 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.674037 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.674040 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.674042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.674138 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.675045 (+ 907us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:54.675051 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:54.675157 (+ 106us) client_negotiation.cc:769] Sending connection context
0807 06:22:54.675279 (+ 122us) client_negotiation.cc:240] Negotiation successful
0807 06:22:54.675464 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":28,"spinlock_wait_cycles":38656,"thread_start_us":209,"threads_started":1}
I20260807 06:22:54.675971 26550 heartbeater.cc:347] Connected to a master server at 127.15.254.62:41399
I20260807 06:22:54.676281 26550 heartbeater.cc:464] Registering TS with master...
Aug 07 06:22:54 dist-test-slave-44d2 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083774, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.60@KRBTEST.COM
I20260807 06:22:54.677039 26550 heartbeater.cc:511] Master 127.15.254.62:41399 requested a full tablet report, sending...
I20260807 06:22:54.677307 26213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.649163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:50859 (local address 127.15.254.61:36519)
0807 06:22:54.649342 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:54.649349 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:54.650894 (+ 1545us) server_negotiation.cc:407] Connection header received
0807 06:22:54.654434 (+ 3540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:54.654439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:54.654534 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:54.654594 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:54.656392 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.657350 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.659648 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.659882 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.659998 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.671117 (+ 11119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:54.671125 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:54.671131 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:54.671171 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:54.673297 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.673841 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.673844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.673846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.673935 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.674601 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.674608 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.674612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.674873 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:54.674985 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:54.676805 (+ 1820us) server_negotiation.cc:299] Negotiation successful
0807 06:22:54.676979 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 06:22:54.678249 26552 heartbeater.cc:347] Connected to a master server at 127.15.254.61:36519
I20260807 06:22:54.678355 26552 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:54.678514 25193 ts_manager.cc:194] Re-registered known tserver with Master: dc9fbde76a594443bf8b835f3b566d63 (127.15.254.1:42279)
I20260807 06:22:54.678563 26552 heartbeater.cc:511] Master 127.15.254.61:36519 requested a full tablet report, sending...
I20260807 06:22:54.679378 25193 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.254.1@KRBTEST.COM'} at 127.15.254.1:42581
I20260807 06:22:54.680270 26042 ts_manager.cc:194] Registered new tserver with Master: dc9fbde76a594443bf8b835f3b566d63 (127.15.254.1:42279)
I20260807 06:22:54.681804 26556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.649554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:44007 (local address 127.15.254.1:56545)
0807 06:22:54.650641 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:54.650705 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:54.654019 (+ 3314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:54.671467 (+ 17448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:54.671470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:54.671665 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:54.671764 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.671770 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.673190 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.673195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.674041 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.674050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.674720 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.674724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.674821 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.675372 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:54.675386 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:54.676905 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:54.679898 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.679902 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.679904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.680169 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.680490 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.680494 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.680523 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.680614 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.681233 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:54.681236 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:54.681337 (+ 101us) client_negotiation.cc:769] Sending connection context
0807 06:22:54.681454 (+ 117us) client_negotiation.cc:240] Negotiation successful
0807 06:22:54.681618 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"spinlock_wait_cycles":558336,"thread_start_us":233,"threads_started":1}
I20260807 06:22:54.681818 26555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.648841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:56545 (local address 127.15.254.60:44007)
0807 06:22:54.671237 (+ 22396us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:54.671241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:54.671253 (+ 12us) server_negotiation.cc:407] Connection header received
0807 06:22:54.671293 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:54.671295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:54.671337 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:54.671460 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:54.671901 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.673069 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.674174 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.674549 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.677650 (+ 3101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.677797 (+ 147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:54.677803 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:54.677806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:54.677830 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:54.679768 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.680279 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.680281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.680283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.680353 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.680748 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.680753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.680758 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.681105 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:54.681242 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:54.681519 (+ 277us) server_negotiation.cc:299] Negotiation successful
0807 06:22:54.681647 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22308,"thread_start_us":121,"threads_started":1}
I20260807 06:22:54.682034 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:34939
--local_ip_for_outbound_sockets=127.15.254.2
--tserver_master_addrs=127.15.254.62:41399
--webserver_port=39363
--webserver_interface=127.15.254.2
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.2.keytab
--principal=oryx/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.254.62:41399,127.15.254.61:36519
--tserver_master_addrs=127.15.254.62:41399,127.15.254.61:36519,127.15.254.60:44007 with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:22:54.682598 26549 heartbeater.cc:347] Connected to a master server at 127.15.254.60:44007
I20260807 06:22:54.682714 26549 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:54.682976 26549 heartbeater.cc:511] Master 127.15.254.60:44007 requested a full tablet report, sending...
I20260807 06:22:54.684288 26293 ts_manager.cc:194] Registered new tserver with Master: dc9fbde76a594443bf8b835f3b566d63 (127.15.254.1:42279)
I20260807 06:22:54.723392 25843 heartbeater.cc:503] Master 127.15.254.62:41399 was elected leader, sending a full tablet report...
W20260807 06:22:54.808308 26562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:54.808682 26562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:54.808774 26562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:54.812994 26562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:54.813064 26562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:54.813167 26562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:22:54.819054 26562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.2.keytab
--principal=oryx/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:34939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=39363
--tserver_master_addrs=127.15.254.62:41399,127.15.254.61:36519,127.15.254.60:44007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:54.820250 26562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:54.821439 26562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:54.832013 26568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:54.832046 26572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:54.832013 26567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:54.832302 26562 server_base.cc:1034] running on GCE node
I20260807 06:22:54.832855 26562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:54.833621 26562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:54.834805 26562 hybrid_clock.cc:648] HybridClock initialized: now 1786083774834786 us; error 29 us; skew 500 ppm
Aug 07 06:22:54 dist-test-slave-44d2 krb5kdc[25128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083774, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:54.838308 26562 init.cc:377] Logged in from keytab as oryx/127.15.254.2@KRBTEST.COM (short username oryx)
I20260807 06:22:54.839282 26562 webserver.cc:460] Webserver started at http://127.15.254.2:39363/ using document root <none> and password file <none>
I20260807 06:22:54.840008 26562 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:54.840099 26562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:54.844642 26562 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:22:54.846824 26582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:54.848119 26562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 06:22:54.848222 26562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "f0e00ec4b6044d4ea221ea3da197ad7d"
format_stamp: "Formatted at 2026-08-07 06:22:53 on dist-test-slave-44d2"
server_key: "8950360f6d3a6df586a8320cd35917e1"
server_key_iv: "20fde2503d9912613d4631afefc9bb2e"
server_key_version: "encryptionkey@0"
I20260807 06:22:54.848824 26562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:54.863997 26562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:54.867396 26562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:54.867654 26562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:54.868321 26562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:54.869374 26562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:54.869419 26562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:54.869462 26562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:54.869515 26562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:54.879582 26562 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:34939
I20260807 06:22:54.879603 26740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:34939 every 8 connection(s)
I20260807 06:22:54.880824 26562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:22:54.890884 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 26562
I20260807 06:22:54.891120 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 25654
Aug 07 06:22:54 dist-test-slave-44d2 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083774, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.61@KRBTEST.COM
Aug 07 06:22:54 dist-test-slave-44d2 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083774, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.60@KRBTEST.COM
Aug 07 06:22:54 dist-test-slave-44d2 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083774, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM
I20260807 06:22:54.899909 26213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.882769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:57583 (local address 127.15.254.61:36519)
0807 06:22:54.882930 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:54.882937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:54.883930 (+ 993us) server_negotiation.cc:407] Connection header received
0807 06:22:54.885350 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:54.885354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:54.885412 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:54.885468 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:54.887283 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.888101 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.889831 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.890139 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.890248 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.894824 (+ 4576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:54.894828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:54.894831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:54.894863 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:54.896968 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.897836 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.897840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.897842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.897941 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.898435 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.898438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.898440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.898686 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:54.898777 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:54.899585 (+ 808us) server_negotiation.cc:299] Negotiation successful
0807 06:22:54.899730 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 06:22:54.899909 25249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.882758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:49609 (local address 127.15.254.62:41399)
0807 06:22:54.882896 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:54.882900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:54.883930 (+ 1030us) server_negotiation.cc:407] Connection header received
0807 06:22:54.885350 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:54.885354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:54.885412 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:54.885468 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:54.887282 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.888115 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.889474 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.889712 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.889843 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.894696 (+ 4853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:54.894703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:54.894710 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:54.894749 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:54.896983 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.897836 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.897840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.897842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.897941 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.898435 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.898438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.898440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.898685 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:54.898777 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:54.899600 (+ 823us) server_negotiation.cc:299] Negotiation successful
0807 06:22:54.899730 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 06:22:54.901664 26749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.883231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.2:49609)
0807 06:22:54.883766 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:54.883820 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:54.885093 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:54.885846 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:54.885856 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:54.886374 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:54.887116 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.887126 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.888227 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.888231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.889333 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.889344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.890021 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.890025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.890264 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.892130 (+ 1866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:54.892152 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:54.894456 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:54.897123 (+ 2667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.897130 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.897150 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.897705 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.898064 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.898067 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.898069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.898311 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.898963 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:54.898965 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:54.899338 (+ 373us) client_negotiation.cc:769] Sending connection context
0807 06:22:54.900090 (+ 752us) client_negotiation.cc:240] Negotiation successful
0807 06:22:54.900592 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":114,"spinlock_wait_cycles":227072,"thread_start_us":198,"threads_started":1}
I20260807 06:22:54.903509 26744 heartbeater.cc:347] Connected to a master server at 127.15.254.62:41399
I20260807 06:22:54.903803 26744 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:54.904524 26744 heartbeater.cc:511] Master 127.15.254.62:41399 requested a full tablet report, sending...
I20260807 06:22:54.904894 26555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.884607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:33109 (local address 127.15.254.60:44007)
0807 06:22:54.884891 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:54.884898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:54.884921 (+ 23us) server_negotiation.cc:407] Connection header received
0807 06:22:54.885364 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:54.885369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:54.885427 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:54.885489 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:54.887273 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.887829 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.889848 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.890111 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.890243 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.897966 (+ 7723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:54.897974 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:54.897980 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:54.898032 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:54.900098 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.902519 (+ 2421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.902526 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.902530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.902705 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:54.903308 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:54.903315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:54.903330 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:54.903761 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:54.903977 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:54.904587 (+ 610us) server_negotiation.cc:299] Negotiation successful
0807 06:22:54.904746 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260807 06:22:54.905006 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.884396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:44007 (local address 127.15.254.2:33109)
0807 06:22:54.884676 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:54.884692 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:54.885116 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:54.885852 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:54.885857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:54.886359 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:54.887104 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.887122 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.888017 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.888024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.889702 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.889709 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.890245 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.890249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.890374 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.891084 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:54.891109 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:54.894662 (+ 3553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:54.900365 (+ 5703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.900372 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.900377 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.900764 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.902937 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.902944 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.902949 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.903127 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.904084 (+ 957us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:54.904090 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:54.904283 (+ 193us) client_negotiation.cc:769] Sending connection context
0807 06:22:54.904427 (+ 144us) client_negotiation.cc:240] Negotiation successful
0807 06:22:54.904665 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":233,"spinlock_wait_cycles":55808,"thread_start_us":141,"threads_started":1}
I20260807 06:22:54.905813 25193 ts_manager.cc:194] Re-registered known tserver with Master: f0e00ec4b6044d4ea221ea3da197ad7d (127.15.254.2:34939)
I20260807 06:22:54.906021 26741 heartbeater.cc:347] Connected to a master server at 127.15.254.60:44007
I20260807 06:22:54.906124 26741 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:54.906301 26741 heartbeater.cc:511] Master 127.15.254.60:44007 requested a full tablet report, sending...
I20260807 06:22:54.906620 25193 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.254.2@KRBTEST.COM'} at 127.15.254.2:49609
I20260807 06:22:54.906662 26748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:54.883231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:36519 (local address 127.15.254.2:57583)
0807 06:22:54.883766 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:54.883819 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:54.885093 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:54.885846 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:54.885856 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:54.886374 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:54.887117 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.887126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.888578 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.888582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.889675 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:54.889688 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:54.890464 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:54.890469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:54.890676 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:54.891405 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:54.891424 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:54.894448 (+ 3024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:54.897123 (+ 2675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.897130 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.897150 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.897705 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.898074 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:54.898077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:54.898079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:54.898311 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:54.898862 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:54.898869 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:54.899268 (+ 399us) client_negotiation.cc:769] Sending connection context
0807 06:22:54.905939 (+ 6671us) client_negotiation.cc:240] Negotiation successful
0807 06:22:54.906132 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":233,"spinlock_wait_cycles":74624,"thread_start_us":151,"threads_started":1}
I20260807 06:22:54.907150 26293 ts_manager.cc:194] Registered new tserver with Master: f0e00ec4b6044d4ea221ea3da197ad7d (127.15.254.2:34939)
I20260807 06:22:54.907951 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:43515
--local_ip_for_outbound_sockets=127.15.254.3
--tserver_master_addrs=127.15.254.62:41399
--webserver_port=32961
--webserver_interface=127.15.254.3
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.3.keytab
--principal=oryx/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.254.62:41399,127.15.254.61:36519
--tserver_master_addrs=127.15.254.62:41399,127.15.254.61:36519,127.15.254.60:44007 with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:22:54.909029 26745 heartbeater.cc:347] Connected to a master server at 127.15.254.61:36519
I20260807 06:22:54.909133 26745 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:54.909322 26745 heartbeater.cc:511] Master 127.15.254.61:36519 requested a full tablet report, sending...
I20260807 06:22:54.910264 26042 ts_manager.cc:194] Registered new tserver with Master: f0e00ec4b6044d4ea221ea3da197ad7d (127.15.254.2:34939)
I20260807 06:22:55.030217 26752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:55.025623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:44007 (local address 127.0.0.1:43570)
0807 06:22:55.026091 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:55.026104 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:55.026225 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:55.026561 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:55.026564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:55.026581 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:55.026683 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:55.026688 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.027780 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.027786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:55.028820 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:55.028833 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.029805 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.029808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:55.029900 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:55.029938 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 06:22:55.029996 (+ 58us) client_negotiation.cc:240] Negotiation successful
0807 06:22:55.030065 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":190,"threads_started":1}
I20260807 06:22:55.030243 26555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:55.025804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.15.254.60:44007)
0807 06:22:55.026055 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:55.026060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:55.026208 (+ 148us) server_negotiation.cc:407] Connection header received
0807 06:22:55.026334 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:55.026338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:55.026412 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:55.026538 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:55.026799 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.027634 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.028967 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.029659 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.029784 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:55.029983 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:55.030023 (+ 40us) server_negotiation.cc:299] Negotiation successful
0807 06:22:55.030100 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260807 06:22:55.030993 26308 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260807 06:22:55.031433 26352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f97442df1a34ff6a8a6215460b56d62. Latest consensus state: current_term: 1 leader_uuid: "9f97442df1a34ff6a8a6215460b56d62" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: NON_VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: true } } }
I20260807 06:22:55.031592 26352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [sys.catalog]: This master's current role is: LEARNER
W20260807 06:22:55.038748 26751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:22:55.039076 26751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:22:55.039125 26751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:22:55.043435 26751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:22:55.043514 26751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:22:55.043576 26751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:22:55.049408 26751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.3.keytab
--principal=oryx/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:43515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=32961
--tserver_master_addrs=127.15.254.62:41399,127.15.254.61:36519,127.15.254.60:44007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:22:55.050784 26751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:22:55.052018 26751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:22:55.063601 26764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:22:55.063601 26760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:55.063674 26751 server_base.cc:1034] running on GCE node
W20260807 06:22:55.063614 26759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:22:55.064389 26751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:22:55.065179 26751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:22:55.066363 26751 hybrid_clock.cc:648] HybridClock initialized: now 1786083775066344 us; error 37 us; skew 500 ppm
Aug 07 06:22:55 dist-test-slave-44d2 krb5kdc[25128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083775, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:22:55.069653 26751 init.cc:377] Logged in from keytab as oryx/127.15.254.3@KRBTEST.COM (short username oryx)
I20260807 06:22:55.070590 26751 webserver.cc:460] Webserver started at http://127.15.254.3:32961/ using document root <none> and password file <none>
I20260807 06:22:55.071199 26751 fs_manager.cc:359] Metadata directory not provided
I20260807 06:22:55.071318 26751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:22:55.075697 26751 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:22:55.078547 26774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:55.080016 26751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:22:55.080173 26751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "043fb1b788ed4af18200209e0ac9f5c0"
format_stamp: "Formatted at 2026-08-07 06:22:53 on dist-test-slave-44d2"
server_key: "5960ec65e549585d70803db126a769d6"
server_key_iv: "59100694866f66072cc34cf58f47890e"
server_key_version: "encryptionkey@0"
I20260807 06:22:55.080865 26751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:22:55.094054 26751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:22:55.097266 26751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:22:55.097482 26751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:22:55.098120 26751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:22:55.099056 26751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:22:55.099125 26751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:55.099197 26751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:22:55.099247 26751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:22:55.108695 26751 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:43515
I20260807 06:22:55.108721 26936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:43515 every 8 connection(s)
I20260807 06:22:55.109968 26751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:22:55.116904 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 26751
Aug 07 06:22:55 dist-test-slave-44d2 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083775, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM
Aug 07 06:22:55 dist-test-slave-44d2 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083775, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.60@KRBTEST.COM
Aug 07 06:22:55 dist-test-slave-44d2 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083775, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.61@KRBTEST.COM
I20260807 06:22:55.131429 26555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:55.113946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:50341 (local address 127.15.254.60:44007)
0807 06:22:55.114088 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:55.114093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:55.114255 (+ 162us) server_negotiation.cc:407] Connection header received
0807 06:22:55.114943 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:55.114946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:55.115007 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:55.115078 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:55.117025 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.117928 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.119928 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.120299 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.120381 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:55.126590 (+ 6209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:55.126594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:55.126597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:55.126627 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:55.128524 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:55.129297 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:55.129300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:55.129302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:55.129397 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:55.129886 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:55.129891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:55.129895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:55.130159 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:55.130239 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:55.131064 (+ 825us) server_negotiation.cc:299] Negotiation successful
0807 06:22:55.131226 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 06:22:55.132762 26956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:55.119585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38876 (local address 127.15.254.61:36519)
0807 06:22:55.121461 (+ 1876us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:55.121467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:55.122199 (+ 732us) server_negotiation.cc:407] Connection header received
0807 06:22:55.122388 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:55.122394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:55.122460 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:55.122560 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:55.124005 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.125020 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.126251 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.126628 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.126714 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:55.128575 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:55.128581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:55.128588 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:55.128619 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:55.130456 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:55.130941 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:55.130948 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:55.130952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:55.131047 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:55.131363 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:55.131367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:55.131369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:55.131892 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:55.132003 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:55.132425 (+ 422us) server_negotiation.cc:299] Negotiation successful
0807 06:22:55.132608 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1794,"thread_start_us":82,"threads_started":1}
I20260807 06:22:55.132897 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:55.112632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.3:52869)
0807 06:22:55.113688 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:55.113702 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:55.114508 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:55.115324 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:55.115331 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:55.115981 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:55.116828 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:55.116854 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.118073 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.118079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:55.119706 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:55.119719 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.120611 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.120616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:55.120939 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:55.121980 (+ 1041us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:55.122004 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:55.125195 (+ 3191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:55.130046 (+ 4851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:55.130054 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:55.130059 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:55.130359 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:55.131205 (+ 846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:55.131209 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:55.131211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:55.131302 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:55.131827 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:55.131830 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:55.131934 (+ 104us) client_negotiation.cc:769] Sending connection context
0807 06:22:55.132037 (+ 103us) client_negotiation.cc:240] Negotiation successful
0807 06:22:55.132189 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"mutex_wait_us":260,"spinlock_wait_cycles":275328,"thread_start_us":163,"threads_started":1}
I20260807 06:22:55.133210 26949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:55.113791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:44007 (local address 127.15.254.3:50341)
0807 06:22:55.114144 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:55.114157 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:55.114524 (+ 367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:55.115322 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:55.115330 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:55.115989 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:55.116836 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:55.116854 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.118117 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.118120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:55.119789 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:55.119796 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.120618 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.120622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:55.120940 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:55.122095 (+ 1155us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:55.122115 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:55.126436 (+ 4321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:55.128653 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:55.128663 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:55.128687 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:55.129164 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:55.129530 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:55.129534 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:55.129537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:55.129757 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:55.130335 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:55.130345 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:55.130741 (+ 396us) client_negotiation.cc:769] Sending connection context
0807 06:22:55.131202 (+ 461us) client_negotiation.cc:240] Negotiation successful
0807 06:22:55.131624 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":450,"spinlock_wait_cycles":17536,"thread_start_us":142,"threads_started":1}
I20260807 06:22:55.134212 25249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:55.112055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:52869 (local address 127.15.254.62:41399)
0807 06:22:55.112207 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:55.112212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:55.113802 (+ 1590us) server_negotiation.cc:407] Connection header received
0807 06:22:55.114869 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:55.114873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:55.115026 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:55.115143 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:55.117031 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.117836 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.119870 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.120242 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.120367 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:55.125388 (+ 5021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:55.125395 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:55.125399 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:55.125433 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:55.129903 (+ 4470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:55.130975 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:55.130981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:55.130983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:55.131082 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:55.131410 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:55.131413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:55.131415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:55.131655 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:55.131777 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:55.133912 (+ 2135us) server_negotiation.cc:299] Negotiation successful
0807 06:22:55.134038 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 06:22:55.134611 26960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:55.120678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43580 (local address 127.15.254.60:44007)
0807 06:22:55.121653 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:55.121657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:55.121794 (+ 137us) server_negotiation.cc:407] Connection header received
0807 06:22:55.121929 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:55.121933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:55.123496 (+ 1563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:55.123599 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:55.124531 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.125296 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.126655 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.127068 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.127178 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:55.129742 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:55.129746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:55.129749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:55.129780 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:55.132231 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:55.132774 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:55.132778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:55.132779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:55.132876 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:55.133233 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:55.133238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:55.133239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:55.133477 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:55.133602 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:55.134301 (+ 699us) server_negotiation.cc:299] Negotiation successful
0807 06:22:55.134431 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":890,"thread_start_us":93,"threads_started":1}
I20260807 06:22:55.134698 26939 heartbeater.cc:347] Connected to a master server at 127.15.254.60:44007
I20260807 06:22:55.134968 26939 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:55.135404 26940 heartbeater.cc:347] Connected to a master server at 127.15.254.62:41399
I20260807 06:22:55.135505 26940 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:55.135594 26939 heartbeater.cc:511] Master 127.15.254.60:44007 requested a full tablet report, sending...
I20260807 06:22:55.136219 26945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:55.112631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:36519 (local address 127.15.254.3:49991)
0807 06:22:55.113228 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:55.113288 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:55.114508 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:55.115449 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:55.115453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:55.115983 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:22:55.116838 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:55.116854 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.118117 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.118120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:55.119778 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:55.119789 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.121223 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.121227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:55.121341 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:55.122012 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:22:55.122023 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:22:55.125616 (+ 3593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:22:55.134395 (+ 8779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:55.134400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:55.134402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:55.134661 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:55.135000 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:22:55.135006 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:22:55.135011 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:22:55.135111 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:22:55.135575 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:22:55.135578 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:22:55.135699 (+ 121us) client_negotiation.cc:769] Sending connection context
0807 06:22:55.135819 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 06:22:55.135966 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":296,"spinlock_wait_cycles":768,"thread_start_us":159,"threads_started":1}
I20260807 06:22:55.136541 26940 heartbeater.cc:511] Master 127.15.254.62:41399 requested a full tablet report, sending...
I20260807 06:22:55.136828 26213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:55.112130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:49991 (local address 127.15.254.61:36519)
0807 06:22:55.112300 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:55.112305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:55.113437 (+ 1132us) server_negotiation.cc:407] Connection header received
0807 06:22:55.115268 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:55.115272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:55.115330 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:55.115415 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:55.117025 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.117929 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.119931 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.120298 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.120381 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:55.125775 (+ 5394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:55.125778 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:55.125781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:55.125808 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:55.130824 (+ 5016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:55.134790 (+ 3966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:55.134795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:55.134800 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:55.134890 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:55.135223 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:55.135226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:55.135228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:55.135425 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:55.135526 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:55.136515 (+ 989us) server_negotiation.cc:299] Negotiation successful
0807 06:22:55.136648 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 06:22:55.137459 26941 heartbeater.cc:347] Connected to a master server at 127.15.254.61:36519
I20260807 06:22:55.137542 26293 ts_manager.cc:194] Registered new tserver with Master: 043fb1b788ed4af18200209e0ac9f5c0 (127.15.254.3:43515)
I20260807 06:22:55.137568 26941 heartbeater.cc:464] Registering TS with master...
I20260807 06:22:55.137756 26941 heartbeater.cc:511] Master 127.15.254.61:36519 requested a full tablet report, sending...
I20260807 06:22:55.138109 25193 ts_manager.cc:194] Re-registered known tserver with Master: 043fb1b788ed4af18200209e0ac9f5c0 (127.15.254.3:43515)
I20260807 06:22:55.138473 26042 ts_manager.cc:194] Registered new tserver with Master: 043fb1b788ed4af18200209e0ac9f5c0 (127.15.254.3:43515)
I20260807 06:22:55.138825 25193 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.254.3@KRBTEST.COM'} at 127.15.254.3:52869
I20260807 06:22:55.141961 26963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:55.120991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48030 (local address 127.15.254.62:41399)
0807 06:22:55.122043 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:55.122048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:55.122063 (+ 15us) server_negotiation.cc:407] Connection header received
0807 06:22:55.122101 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:55.122105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:55.122162 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:55.122254 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:55.124021 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.124898 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.135187 (+ 10289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.135560 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.135659 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:55.137633 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:55.137638 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:55.137641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:55.137667 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:55.139944 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:55.140410 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:55.140414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:55.140415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:55.140579 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:55.140988 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:55.140992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:55.140993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:55.141173 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:55.141319 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:55.141590 (+ 271us) server_negotiation.cc:299] Negotiation successful
0807 06:22:55.141731 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":965,"thread_start_us":68,"threads_started":1}
I20260807 06:22:55.145038 25193 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:22:55.148129 25193 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:22:55.151207 26243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f97442df1a34ff6a8a6215460b56d62" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: NON_VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: true } } }
I20260807 06:22:55.151364 26243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:55.155994 26967 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62: attempting to promote NON_VOTER 8e73d6f7d63a48379cbc6aa648137096 to VOTER
I20260807 06:22:55.156539 26967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: false } }
I20260807 06:22:55.158012 26308 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 1 LEARNER]: Refusing update from remote peer 9f97442df1a34ff6a8a6215460b56d62: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260807 06:22:55.158044 26045 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 1 FOLLOWER]: Refusing update from remote peer 9f97442df1a34ff6a8a6215460b56d62: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260807 06:22:55.158392 26243 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260807 06:22:55.158564 26966 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260807 06:22:55.160367 26965 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8e73d6f7d63a48379cbc6aa648137096 (127.15.254.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: false } } }
I20260807 06:22:55.160915 26308 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8e73d6f7d63a48379cbc6aa648137096 (127.15.254.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: false } } }
I20260807 06:22:55.161953 26045 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8e73d6f7d63a48379cbc6aa648137096 (127.15.254.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: false } } }
I20260807 06:22:55.162132 26965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9f97442df1a34ff6a8a6215460b56d62" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: false } } }
I20260807 06:22:55.162230 26965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:55.162361 26965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f97442df1a34ff6a8a6215460b56d62" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: false } } }
I20260807 06:22:55.162446 26965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:55.162489 26966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f97442df1a34ff6a8a6215460b56d62" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: false } } }
I20260807 06:22:55.162627 26966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62 [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:55.164366 26352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f97442df1a34ff6a8a6215460b56d62" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: false } } }
I20260807 06:22:55.164548 26352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:22:55.164685 26976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f97442df1a34ff6a8a6215460b56d62" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: false } } }
I20260807 06:22:55.164867 26976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:22:55.173442 26977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:55.167249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:34939 (local address 127.0.0.1:55258)
0807 06:22:55.167618 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:55.167631 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:55.167751 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:55.169229 (+ 1478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:55.169232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:55.169484 (+ 252us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:55.169594 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:55.169601 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.170933 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.170937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:55.171891 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:55.171901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.172873 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.172876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:55.173084 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:55.173112 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 06:22:55.173183 (+ 71us) client_negotiation.cc:240] Negotiation successful
0807 06:22:55.173257 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":176,"spinlock_wait_cycles":511360,"thread_start_us":201,"threads_started":1}
I20260807 06:22:55.173450 26981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:55.168542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39156 (local address 127.15.254.1:42279)
0807 06:22:55.168823 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:55.168838 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:55.168867 (+ 29us) server_negotiation.cc:407] Connection header received
0807 06:22:55.168935 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:55.168939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:55.169088 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:55.169254 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:55.169467 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.170359 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.171772 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.172706 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.172829 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:55.173002 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:55.173183 (+ 181us) server_negotiation.cc:299] Negotiation successful
0807 06:22:55.173282 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":102,"threads_started":1}
I20260807 06:22:55.173494 26980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:55.168094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55258 (local address 127.15.254.2:34939)
0807 06:22:55.168605 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:55.168617 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:55.168640 (+ 23us) server_negotiation.cc:407] Connection header received
0807 06:22:55.168815 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:55.168822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:55.169030 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:55.169311 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:55.169724 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.170803 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.172065 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.172726 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.172975 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:55.173152 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:55.173209 (+ 57us) server_negotiation.cc:299] Negotiation successful
0807 06:22:55.173309 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":139,"threads_started":1}
I20260807 06:22:55.174254 26752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:55.166981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:43515 (local address 127.0.0.1:42090)
0807 06:22:55.167183 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:55.167200 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:55.167280 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:55.168584 (+ 1304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:55.168587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:55.168608 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:55.168726 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:55.168733 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.170317 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.170320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:55.171557 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:55.171569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.172851 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.172855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:55.173963 (+ 1108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:55.173985 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 06:22:55.174055 (+ 70us) client_negotiation.cc:240] Negotiation successful
0807 06:22:55.174131 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":964}
I20260807 06:22:55.174396 26978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:55.167514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42090 (local address 127.15.254.3:43515)
0807 06:22:55.167910 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:55.167926 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:55.167952 (+ 26us) server_negotiation.cc:407] Connection header received
0807 06:22:55.168045 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:55.168054 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:55.168344 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:55.168893 (+ 549us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:55.169037 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.170193 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.171709 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.172712 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.173624 (+ 912us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:55.173789 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:55.174122 (+ 333us) server_negotiation.cc:299] Negotiation successful
0807 06:22:55.174234 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":131,"threads_started":1}
I20260807 06:22:55.174701 26979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:55.167732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:42279 (local address 127.0.0.1:39156)
0807 06:22:55.167984 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:55.168005 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:55.168095 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:55.169228 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:55.169231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:55.169263 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:55.169362 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:55.169371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.170498 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.170501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:55.171630 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:55.171640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.172844 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.172851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:55.173004 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:55.173045 (+ 41us) client_negotiation.cc:769] Sending connection context
0807 06:22:55.173126 (+ 81us) client_negotiation.cc:240] Negotiation successful
0807 06:22:55.174582 (+ 1456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":36,"thread_start_us":120,"threads_started":1}
I20260807 06:22:55.176852 26652 tablet_service.cc:1467] Processing CreateTablet for tablet 4f894b906af044df904d6542700f5ca8 (DEFAULT_TABLE table=test-table [id=443919bfa91045d49b9af353d1e30eda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:55.177178 26845 tablet_service.cc:1467] Processing CreateTablet for tablet 4f894b906af044df904d6542700f5ca8 (DEFAULT_TABLE table=test-table [id=443919bfa91045d49b9af353d1e30eda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:55.178054 26652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f894b906af044df904d6542700f5ca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:55.178267 26845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f894b906af044df904d6542700f5ca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:55.178115 26454 tablet_service.cc:1467] Processing CreateTablet for tablet 4f894b906af044df904d6542700f5ca8 (DEFAULT_TABLE table=test-table [id=443919bfa91045d49b9af353d1e30eda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:55.179111 26454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f894b906af044df904d6542700f5ca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:55.186448 26982 tablet_bootstrap.cc:492] T 4f894b906af044df904d6542700f5ca8 P f0e00ec4b6044d4ea221ea3da197ad7d: Bootstrap starting.
I20260807 06:22:55.187317 26986 tablet_bootstrap.cc:492] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63: Bootstrap starting.
I20260807 06:22:55.187846 26985 tablet_bootstrap.cc:492] T 4f894b906af044df904d6542700f5ca8 P 043fb1b788ed4af18200209e0ac9f5c0: Bootstrap starting.
I20260807 06:22:55.189224 26982 tablet_bootstrap.cc:654] T 4f894b906af044df904d6542700f5ca8 P f0e00ec4b6044d4ea221ea3da197ad7d: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:55.189644 26986 tablet_bootstrap.cc:654] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:55.189836 26985 tablet_bootstrap.cc:654] T 4f894b906af044df904d6542700f5ca8 P 043fb1b788ed4af18200209e0ac9f5c0: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:55.190412 26982 log.cc:824] T 4f894b906af044df904d6542700f5ca8 P f0e00ec4b6044d4ea221ea3da197ad7d: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:55.190548 26986 log.cc:824] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:55.190891 26985 log.cc:824] T 4f894b906af044df904d6542700f5ca8 P 043fb1b788ed4af18200209e0ac9f5c0: Log is configured to *not* fsync() on all Append() calls
I20260807 06:22:55.192814 26986 tablet_bootstrap.cc:492] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63: No bootstrap required, opened a new log
I20260807 06:22:55.192862 26982 tablet_bootstrap.cc:492] T 4f894b906af044df904d6542700f5ca8 P f0e00ec4b6044d4ea221ea3da197ad7d: No bootstrap required, opened a new log
I20260807 06:22:55.193060 26986 ts_tablet_manager.cc:1397] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260807 06:22:55.193055 26985 tablet_bootstrap.cc:492] T 4f894b906af044df904d6542700f5ca8 P 043fb1b788ed4af18200209e0ac9f5c0: No bootstrap required, opened a new log
I20260807 06:22:55.193071 26982 ts_tablet_manager.cc:1397] T 4f894b906af044df904d6542700f5ca8 P f0e00ec4b6044d4ea221ea3da197ad7d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260807 06:22:55.193255 26985 ts_tablet_manager.cc:1397] T 4f894b906af044df904d6542700f5ca8 P 043fb1b788ed4af18200209e0ac9f5c0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 06:22:55.196970 26986 raft_consensus.cc:348] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "043fb1b788ed4af18200209e0ac9f5c0" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43515 } } peers { permanent_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34939 } } peers { permanent_uuid: "dc9fbde76a594443bf8b835f3b566d63" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42279 } }
I20260807 06:22:55.197284 26986 raft_consensus.cc:374] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:55.197342 26986 raft_consensus.cc:729] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9fbde76a594443bf8b835f3b566d63, State: Initialized, Role: FOLLOWER
I20260807 06:22:55.197873 26986 consensus_queue.cc:260] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "043fb1b788ed4af18200209e0ac9f5c0" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43515 } } peers { permanent_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34939 } } peers { permanent_uuid: "dc9fbde76a594443bf8b835f3b566d63" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42279 } }
I20260807 06:22:55.198534 26985 raft_consensus.cc:348] T 4f894b906af044df904d6542700f5ca8 P 043fb1b788ed4af18200209e0ac9f5c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "043fb1b788ed4af18200209e0ac9f5c0" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43515 } } peers { permanent_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34939 } } peers { permanent_uuid: "dc9fbde76a594443bf8b835f3b566d63" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42279 } }
I20260807 06:22:55.198534 26982 raft_consensus.cc:348] T 4f894b906af044df904d6542700f5ca8 P f0e00ec4b6044d4ea221ea3da197ad7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "043fb1b788ed4af18200209e0ac9f5c0" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43515 } } peers { permanent_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34939 } } peers { permanent_uuid: "dc9fbde76a594443bf8b835f3b566d63" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42279 } }
I20260807 06:22:55.198807 26982 raft_consensus.cc:374] T 4f894b906af044df904d6542700f5ca8 P f0e00ec4b6044d4ea221ea3da197ad7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:55.198805 26985 raft_consensus.cc:374] T 4f894b906af044df904d6542700f5ca8 P 043fb1b788ed4af18200209e0ac9f5c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:55.198858 26985 raft_consensus.cc:729] T 4f894b906af044df904d6542700f5ca8 P 043fb1b788ed4af18200209e0ac9f5c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 043fb1b788ed4af18200209e0ac9f5c0, State: Initialized, Role: FOLLOWER
I20260807 06:22:55.198858 26982 raft_consensus.cc:729] T 4f894b906af044df904d6542700f5ca8 P f0e00ec4b6044d4ea221ea3da197ad7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e00ec4b6044d4ea221ea3da197ad7d, State: Initialized, Role: FOLLOWER
I20260807 06:22:55.199118 26986 ts_tablet_manager.cc:1428] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260807 06:22:55.199363 26982 consensus_queue.cc:260] T 4f894b906af044df904d6542700f5ca8 P f0e00ec4b6044d4ea221ea3da197ad7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "043fb1b788ed4af18200209e0ac9f5c0" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43515 } } peers { permanent_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34939 } } peers { permanent_uuid: "dc9fbde76a594443bf8b835f3b566d63" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42279 } }
I20260807 06:22:55.199542 26550 heartbeater.cc:503] Master 127.15.254.62:41399 was elected leader, sending a full tablet report...
I20260807 06:22:55.200518 26982 ts_tablet_manager.cc:1428] T 4f894b906af044df904d6542700f5ca8 P f0e00ec4b6044d4ea221ea3da197ad7d: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20260807 06:22:55.199363 26985 consensus_queue.cc:260] T 4f894b906af044df904d6542700f5ca8 P 043fb1b788ed4af18200209e0ac9f5c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "043fb1b788ed4af18200209e0ac9f5c0" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43515 } } peers { permanent_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34939 } } peers { permanent_uuid: "dc9fbde76a594443bf8b835f3b566d63" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42279 } }
I20260807 06:22:55.200942 26744 heartbeater.cc:503] Master 127.15.254.62:41399 was elected leader, sending a full tablet report...
I20260807 06:22:55.202617 26985 ts_tablet_manager.cc:1428] T 4f894b906af044df904d6542700f5ca8 P 043fb1b788ed4af18200209e0ac9f5c0: Time spent starting tablet: real 0.009s user 0.004s sys 0.003s
I20260807 06:22:55.204367 26940 heartbeater.cc:503] Master 127.15.254.62:41399 was elected leader, sending a full tablet report...
W20260807 06:22:55.362274 26942 tablet.cc:2369] T 4f894b906af044df904d6542700f5ca8 P 043fb1b788ed4af18200209e0ac9f5c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:22:55.374006 26992 raft_consensus.cc:482] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:22:55.374269 26992 raft_consensus.cc:504] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "043fb1b788ed4af18200209e0ac9f5c0" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43515 } } peers { permanent_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34939 } } peers { permanent_uuid: "dc9fbde76a594443bf8b835f3b566d63" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42279 } }
I20260807 06:22:55.375634 26992 leader_election.cc:288] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 043fb1b788ed4af18200209e0ac9f5c0 (127.15.254.3:43515), f0e00ec4b6044d4ea221ea3da197ad7d (127.15.254.2:34939)
I20260807 06:22:55.380506 26995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:55.375851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:43515 (local address 127.15.254.1:58435)
0807 06:22:55.376585 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:55.376608 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:55.376768 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:55.377088 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:55.377092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:55.377113 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:55.377293 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:55.377301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.378187 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.378191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:55.379180 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:55.379191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.379889 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.379894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:55.380077 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:55.380105 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 06:22:55.380193 (+ 88us) client_negotiation.cc:240] Negotiation successful
0807 06:22:55.380260 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":137,"threads_started":1}
I20260807 06:22:55.380529 26978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:55.376012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:58435 (local address 127.15.254.3:43515)
0807 06:22:55.376196 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:55.376227 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:55.376722 (+ 495us) server_negotiation.cc:407] Connection header received
0807 06:22:55.376894 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:55.376900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:55.376971 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:55.377059 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:55.377458 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.378062 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.379321 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.379750 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.379848 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:55.379895 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:55.380246 (+ 351us) server_negotiation.cc:299] Negotiation successful
0807 06:22:55.380337 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 06:22:55.380618 26980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:55.376304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:52339 (local address 127.15.254.2:34939)
0807 06:22:55.376580 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:55.376584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:55.376997 (+ 413us) server_negotiation.cc:407] Connection header received
0807 06:22:55.377090 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:55.377093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:55.377144 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:55.377235 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:55.377586 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.378178 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.379408 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.379905 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.380041 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:55.380098 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:55.380344 (+ 246us) server_negotiation.cc:299] Negotiation successful
0807 06:22:55.380427 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 06:22:55.380590 26996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:55.375996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:34939 (local address 127.15.254.1:52339)
0807 06:22:55.376793 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:55.376804 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:55.377005 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:55.377284 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:55.377286 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:55.377305 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:55.377452 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:55.377459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.378301 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.378304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:55.379239 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:55.379278 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.380048 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.380051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:55.380153 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:55.380213 (+ 60us) client_negotiation.cc:769] Sending connection context
0807 06:22:55.380300 (+ 87us) client_negotiation.cc:240] Negotiation successful
0807 06:22:55.380369 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":585,"threads_started":1}
I20260807 06:22:55.381572 26871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f894b906af044df904d6542700f5ca8" candidate_uuid: "dc9fbde76a594443bf8b835f3b566d63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "043fb1b788ed4af18200209e0ac9f5c0" is_pre_election: true
I20260807 06:22:55.381572 26676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f894b906af044df904d6542700f5ca8" candidate_uuid: "dc9fbde76a594443bf8b835f3b566d63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" is_pre_election: true
I20260807 06:22:55.381930 26676 raft_consensus.cc:2454] T 4f894b906af044df904d6542700f5ca8 P f0e00ec4b6044d4ea221ea3da197ad7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc9fbde76a594443bf8b835f3b566d63 in term 0.
I20260807 06:22:55.381932 26871 raft_consensus.cc:2454] T 4f894b906af044df904d6542700f5ca8 P 043fb1b788ed4af18200209e0ac9f5c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc9fbde76a594443bf8b835f3b566d63 in term 0.
I20260807 06:22:55.382467 26390 leader_election.cc:302] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc9fbde76a594443bf8b835f3b566d63, f0e00ec4b6044d4ea221ea3da197ad7d; no voters:
I20260807 06:22:55.382793 26992 raft_consensus.cc:2781] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:22:55.382884 26992 raft_consensus.cc:482] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:22:55.382943 26992 raft_consensus.cc:3037] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63 [term 0 FOLLOWER]: Advancing to term 1
W20260807 06:22:55.383665 26746 tablet.cc:2369] T 4f894b906af044df904d6542700f5ca8 P f0e00ec4b6044d4ea221ea3da197ad7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:22:55.384433 26992 raft_consensus.cc:504] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "043fb1b788ed4af18200209e0ac9f5c0" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43515 } } peers { permanent_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34939 } } peers { permanent_uuid: "dc9fbde76a594443bf8b835f3b566d63" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42279 } }
I20260807 06:22:55.384929 26992 leader_election.cc:288] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63 [CANDIDATE]: Term 1 election: Requested vote from peers 043fb1b788ed4af18200209e0ac9f5c0 (127.15.254.3:43515), f0e00ec4b6044d4ea221ea3da197ad7d (127.15.254.2:34939)
I20260807 06:22:55.385437 26871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f894b906af044df904d6542700f5ca8" candidate_uuid: "dc9fbde76a594443bf8b835f3b566d63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "043fb1b788ed4af18200209e0ac9f5c0"
I20260807 06:22:55.385437 26676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f894b906af044df904d6542700f5ca8" candidate_uuid: "dc9fbde76a594443bf8b835f3b566d63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d"
I20260807 06:22:55.385617 26676 raft_consensus.cc:3037] T 4f894b906af044df904d6542700f5ca8 P f0e00ec4b6044d4ea221ea3da197ad7d [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:55.385592 26871 raft_consensus.cc:3037] T 4f894b906af044df904d6542700f5ca8 P 043fb1b788ed4af18200209e0ac9f5c0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:55.387142 26871 raft_consensus.cc:2454] T 4f894b906af044df904d6542700f5ca8 P 043fb1b788ed4af18200209e0ac9f5c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc9fbde76a594443bf8b835f3b566d63 in term 1.
I20260807 06:22:55.387142 26676 raft_consensus.cc:2454] T 4f894b906af044df904d6542700f5ca8 P f0e00ec4b6044d4ea221ea3da197ad7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc9fbde76a594443bf8b835f3b566d63 in term 1.
I20260807 06:22:55.387506 26390 leader_election.cc:302] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc9fbde76a594443bf8b835f3b566d63, f0e00ec4b6044d4ea221ea3da197ad7d; no voters:
I20260807 06:22:55.387777 26992 raft_consensus.cc:2781] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:55.388042 26992 raft_consensus.cc:686] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63 [term 1 LEADER]: Becoming Leader. State: Replica: dc9fbde76a594443bf8b835f3b566d63, State: Running, Role: LEADER
I20260807 06:22:55.388433 26992 consensus_queue.cc:237] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "043fb1b788ed4af18200209e0ac9f5c0" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43515 } } peers { permanent_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34939 } } peers { permanent_uuid: "dc9fbde76a594443bf8b835f3b566d63" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42279 } }
I20260807 06:22:55.392727 25192 catalog_manager.cc:5495] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc9fbde76a594443bf8b835f3b566d63 (127.15.254.1). New cstate: current_term: 1 leader_uuid: "dc9fbde76a594443bf8b835f3b566d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "043fb1b788ed4af18200209e0ac9f5c0" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fbde76a594443bf8b835f3b566d63" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42279 } health_report { overall_health: HEALTHY } } }
W20260807 06:22:55.399376 26553 tablet.cc:2369] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:22:55.417032 26981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:55.413147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39158 (local address 127.15.254.1:42279)
0807 06:22:55.413302 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:55.413308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:55.413324 (+ 16us) server_negotiation.cc:407] Connection header received
0807 06:22:55.413533 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:55.413535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:55.413596 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:55.413715 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:22:55.413971 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.414493 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.415364 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.415594 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.415709 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:55.416046 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:22:55.416570 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:22:55.416714 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:55.416776 (+ 62us) server_negotiation.cc:299] Negotiation successful
0807 06:22:55.416859 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 06:22:55.423547 26676 raft_consensus.cc:1261] T 4f894b906af044df904d6542700f5ca8 P f0e00ec4b6044d4ea221ea3da197ad7d [term 1 FOLLOWER]: Refusing update from remote peer dc9fbde76a594443bf8b835f3b566d63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:22:55.423547 26871 raft_consensus.cc:1261] T 4f894b906af044df904d6542700f5ca8 P 043fb1b788ed4af18200209e0ac9f5c0 [term 1 FOLLOWER]: Refusing update from remote peer dc9fbde76a594443bf8b835f3b566d63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:22:55.424412 26997 consensus_queue.cc:1035] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:22:55.424682 26992 consensus_queue.cc:1035] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "043fb1b788ed4af18200209e0ac9f5c0" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:22:55.444921 25191 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:22:55.445127 25191 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:22:55.450407 25191 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f97442df1a34ff6a8a6215460b56d62: Sending DeleteTablet for 3 replicas of tablet 4f894b906af044df904d6542700f5ca8
I20260807 06:22:55.451284 26652 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f894b906af044df904d6542700f5ca8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:55 UTC) from {username='oryx', principal='oryx/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:55258
I20260807 06:22:55.451292 26845 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f894b906af044df904d6542700f5ca8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:55 UTC) from {username='oryx', principal='oryx/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:42090
I20260807 06:22:55.451584 26454 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f894b906af044df904d6542700f5ca8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:55 UTC) from {username='oryx', principal='oryx/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:39156
I20260807 06:22:55.451906 27016 tablet_replica.cc:330] T 4f894b906af044df904d6542700f5ca8 P 043fb1b788ed4af18200209e0ac9f5c0: stopping tablet replica
I20260807 06:22:55.452257 27020 tablet_replica.cc:330] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63: stopping tablet replica
I20260807 06:22:55.452567 27020 raft_consensus.cc:2229] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:22:55.453011 27020 raft_consensus.cc:2258] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:55.452404 27016 raft_consensus.cc:2229] T 4f894b906af044df904d6542700f5ca8 P 043fb1b788ed4af18200209e0ac9f5c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:22:55.453588 27014 tablet_replica.cc:330] T 4f894b906af044df904d6542700f5ca8 P f0e00ec4b6044d4ea221ea3da197ad7d: stopping tablet replica
I20260807 06:22:55.453859 27016 raft_consensus.cc:2258] T 4f894b906af044df904d6542700f5ca8 P 043fb1b788ed4af18200209e0ac9f5c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:55.453935 27014 raft_consensus.cc:2229] T 4f894b906af044df904d6542700f5ca8 P f0e00ec4b6044d4ea221ea3da197ad7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:22:55.454262 27014 raft_consensus.cc:2258] T 4f894b906af044df904d6542700f5ca8 P f0e00ec4b6044d4ea221ea3da197ad7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:55.454440 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 25149
I20260807 06:22:55.460799 27016 ts_tablet_manager.cc:1905] T 4f894b906af044df904d6542700f5ca8 P 043fb1b788ed4af18200209e0ac9f5c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:22:55.464156 27016 ts_tablet_manager.cc:1918] T 4f894b906af044df904d6542700f5ca8 P 043fb1b788ed4af18200209e0ac9f5c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:22:55.464167 27014 ts_tablet_manager.cc:1905] T 4f894b906af044df904d6542700f5ca8 P f0e00ec4b6044d4ea221ea3da197ad7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:22:55.464419 27016 log.cc:1196] T 4f894b906af044df904d6542700f5ca8 P 043fb1b788ed4af18200209e0ac9f5c0: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/4f894b906af044df904d6542700f5ca8
I20260807 06:22:55.464993 27016 ts_tablet_manager.cc:1939] T 4f894b906af044df904d6542700f5ca8 P 043fb1b788ed4af18200209e0ac9f5c0: Deleting consensus metadata
I20260807 06:22:55.468149 27014 ts_tablet_manager.cc:1918] T 4f894b906af044df904d6542700f5ca8 P f0e00ec4b6044d4ea221ea3da197ad7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:22:55.468284 27014 log.cc:1196] T 4f894b906af044df904d6542700f5ca8 P f0e00ec4b6044d4ea221ea3da197ad7d: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/4f894b906af044df904d6542700f5ca8
I20260807 06:22:55.468695 27014 ts_tablet_manager.cc:1939] T 4f894b906af044df904d6542700f5ca8 P f0e00ec4b6044d4ea221ea3da197ad7d: Deleting consensus metadata
I20260807 06:22:55.471397 27020 ts_tablet_manager.cc:1905] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260807 06:22:55.472344 26583 connection.cc:535] server connection from 127.0.0.1:55258 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260807 06:22:55.472841 26780 connection.cc:535] server connection from 127.0.0.1:42090 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 06:22:55.475067 27020 ts_tablet_manager.cc:1918] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:22:55.475198 27020 log.cc:1196] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-0/wal/wals/4f894b906af044df904d6542700f5ca8
I20260807 06:22:55.475572 27020 ts_tablet_manager.cc:1939] T 4f894b906af044df904d6542700f5ca8 P dc9fbde76a594443bf8b835f3b566d63: Deleting consensus metadata
W20260807 06:22:55.476929 26390 connection.cc:441] server connection from 127.0.0.1:39156 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39156 (request call id 1) could send its response
I20260807 06:22:55.478926 26213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:55.452912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38882 (local address 127.15.254.61:36519)
0807 06:22:55.453058 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:55.453065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:55.466800 (+ 13735us) server_negotiation.cc:407] Connection header received
0807 06:22:55.467047 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:55.467050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:55.467107 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:55.467161 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:55.468441 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.469321 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.470630 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.470986 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.471081 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:55.474159 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:55.474164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:55.474167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:55.474205 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:55.476280 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:55.476948 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:55.476952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:55.476954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:55.477052 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:55.477766 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:55.477770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:55.477772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:55.478069 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:55.478200 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:55.478565 (+ 365us) server_negotiation.cc:299] Negotiation successful
0807 06:22:55.478707 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 06:22:55.487926 26956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:55.477569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38890 (local address 127.15.254.61:36519)
0807 06:22:55.477703 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:55.477707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:55.477990 (+ 283us) server_negotiation.cc:407] Connection header received
0807 06:22:55.478176 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:55.478179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:55.478231 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:55.478280 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:55.479201 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.479953 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.481408 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.481762 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.481843 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:55.483758 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:55.483761 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:55.483764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:55.483795 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:55.485788 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:55.486475 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:55.486479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:55.486481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:55.486571 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:55.486922 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:55.486926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:55.486928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:55.487116 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:55.487262 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:55.487567 (+ 305us) server_negotiation.cc:299] Negotiation successful
0807 06:22:55.487715 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 06:22:55.490437 26555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:55.478190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43598 (local address 127.15.254.60:44007)
0807 06:22:55.478313 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:55.478317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:55.479347 (+ 1030us) server_negotiation.cc:407] Connection header received
0807 06:22:55.479525 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:55.479529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:55.479580 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:55.479630 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:55.480772 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.481571 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.483108 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.483456 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.483530 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:55.485590 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:55.485594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:55.485597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:55.485623 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:55.488515 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:55.488971 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:55.488976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:55.488979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:55.489089 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:55.489526 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:55.489530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:55.489532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:55.489746 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:55.489882 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:55.490129 (+ 247us) server_negotiation.cc:299] Negotiation successful
0807 06:22:55.490247 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 06:22:55.490764 26960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:55.453223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43596 (local address 127.15.254.60:44007)
0807 06:22:55.453395 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:55.453409 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:55.480429 (+ 27020us) server_negotiation.cc:407] Connection header received
0807 06:22:55.480641 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:55.480644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:55.480696 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:55.480748 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:22:55.481597 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.482319 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.483549 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:55.483906 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:55.483996 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:55.486439 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:22:55.486446 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:22:55.486452 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:22:55.486481 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:22:55.488317 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:55.489341 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:55.489344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:55.489346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:55.489429 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:22:55.489739 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:22:55.489743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:22:55.489745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:22:55.489980 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:22:55.490109 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:55.490443 (+ 334us) server_negotiation.cc:299] Negotiation successful
0807 06:22:55.490583 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 06:22:56.438403 27036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:56.437754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.3:40927)
0807 06:22:56.438154 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:56.438262 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":174,"threads_started":1}
W20260807 06:22:56.438822 26940 heartbeater.cc:644] Failed to heartbeat to 127.15.254.62:41399 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
I20260807 06:22:56.439924 27037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:56.439437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.1:55521)
0807 06:22:56.439711 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:56.439817 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":110,"threads_started":1}
W20260807 06:22:56.440276 26550 heartbeater.cc:644] Failed to heartbeat to 127.15.254.62:41399 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
I20260807 06:22:56.441504 27038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:56.440968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.2:46291)
0807 06:22:56.441255 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:56.441396 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":114,"threads_started":1}
W20260807 06:22:56.441746 26744 heartbeater.cc:644] Failed to heartbeat to 127.15.254.62:41399 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
I20260807 06:22:56.950584 27039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9f97442df1a34ff6a8a6215460b56d62)
I20260807 06:22:56.950850 27039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: false } }
I20260807 06:22:56.951660 27040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9f97442df1a34ff6a8a6215460b56d62)
I20260807 06:22:56.951870 27040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: false } }
I20260807 06:22:56.952255 27039 leader_election.cc:288] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f97442df1a34ff6a8a6215460b56d62 (127.15.254.62:41399), 8e73d6f7d63a48379cbc6aa648137096 (127.15.254.60:44007)
I20260807 06:22:56.953006 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:56.952430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.0.0.1:48504)
0807 06:22:56.952777 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:56.952882 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":164,"threads_started":1}
W20260807 06:22:56.953222 26020 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
I20260807 06:22:56.953635 27043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:56.953130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.0.0.1:48516)
0807 06:22:56.953435 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:56.953521 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":115,"threads_started":1}
I20260807 06:22:56.953869 27040 leader_election.cc:288] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f97442df1a34ff6a8a6215460b56d62 (127.15.254.62:41399), 5d972ddbeae549fba9841504b5db758a (127.15.254.61:36519)
W20260807 06:22:56.953918 26275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
I20260807 06:22:56.954209 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:56.953886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.0.0.1:48528)
0807 06:22:56.954069 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:56.954124 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260807 06:22:56.954331 26020 leader_election.cc:334] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9f97442df1a34ff6a8a6215460b56d62 (127.15.254.62:41399): Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
I20260807 06:22:56.955246 27053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:56.954703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.0.0.1:48544)
0807 06:22:56.955079 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:56.955137 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":123,"threads_started":1}
W20260807 06:22:56.955365 26275 leader_election.cc:334] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9f97442df1a34ff6a8a6215460b56d62 (127.15.254.62:41399): Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
I20260807 06:22:56.958274 27045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:56.953420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43608 (local address 127.15.254.60:44007)
0807 06:22:56.953721 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:56.953728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:56.953744 (+ 16us) server_negotiation.cc:407] Connection header received
0807 06:22:56.954531 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:56.954536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:56.954591 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:56.954983 (+ 392us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:56.955085 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:56.955851 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:56.957137 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:56.957577 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:56.957708 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:56.957775 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:56.958026 (+ 251us) server_negotiation.cc:299] Negotiation successful
0807 06:22:56.958104 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":71,"threads_started":1}
I20260807 06:22:56.958288 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:56.952895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:44007 (local address 127.0.0.1:43608)
0807 06:22:56.953173 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:56.953188 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:56.953305 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:56.954746 (+ 1441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:56.954749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:56.954769 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:56.954859 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:56.954866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:56.955959 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:56.955963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:56.957003 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:56.957016 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:56.957716 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:56.957720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:56.957871 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:56.957908 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 06:22:56.958006 (+ 98us) client_negotiation.cc:240] Negotiation successful
0807 06:22:56.958094 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":36,"thread_start_us":57,"threads_started":1}
I20260807 06:22:56.958304 27052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:56.954065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38904 (local address 127.15.254.61:36519)
0807 06:22:56.954357 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:56.954366 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:56.954381 (+ 15us) server_negotiation.cc:407] Connection header received
0807 06:22:56.954512 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:56.954516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:56.954572 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:56.954667 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:56.955003 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:56.955866 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:56.957234 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:56.957694 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:56.957890 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:56.957955 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:56.958079 (+ 124us) server_negotiation.cc:299] Negotiation successful
0807 06:22:56.958153 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":221,"thread_start_us":105,"threads_started":1}
I20260807 06:22:56.958302 27043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:56.953829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:36519 (local address 127.0.0.1:38904)
0807 06:22:56.954146 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:56.954160 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:56.954360 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:56.954674 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:56.954679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:56.954737 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:56.954869 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:56.954876 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:56.955976 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:56.955980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:56.957101 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:56.957113 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:56.957832 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:56.957837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:56.957947 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:56.957979 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 06:22:56.958062 (+ 83us) client_negotiation.cc:240] Negotiation successful
0807 06:22:56.958143 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":24}
I20260807 06:22:56.958890 26045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8e73d6f7d63a48379cbc6aa648137096" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5d972ddbeae549fba9841504b5db758a" is_pre_election: true
I20260807 06:22:56.959028 26308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d972ddbeae549fba9841504b5db758a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8e73d6f7d63a48379cbc6aa648137096" is_pre_election: true
I20260807 06:22:56.959183 26045 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e73d6f7d63a48379cbc6aa648137096 in term 1.
I20260807 06:22:56.959280 26308 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d972ddbeae549fba9841504b5db758a in term 1.
I20260807 06:22:56.959627 26270 leader_election.cc:302] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d972ddbeae549fba9841504b5db758a, 8e73d6f7d63a48379cbc6aa648137096; no voters: 9f97442df1a34ff6a8a6215460b56d62
I20260807 06:22:56.959650 26020 leader_election.cc:302] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d972ddbeae549fba9841504b5db758a, 8e73d6f7d63a48379cbc6aa648137096; no voters: 9f97442df1a34ff6a8a6215460b56d62
I20260807 06:22:56.959889 27040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 06:22:56.959892 27039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 06:22:56.959949 27039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9f97442df1a34ff6a8a6215460b56d62)
I20260807 06:22:56.959949 27040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9f97442df1a34ff6a8a6215460b56d62)
I20260807 06:22:56.959988 27039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 1 FOLLOWER]: Advancing to term 2
I20260807 06:22:56.959992 27040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 1 FOLLOWER]: Advancing to term 2
I20260807 06:22:56.961448 27040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: false } }
I20260807 06:22:56.961453 27039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: false } }
I20260807 06:22:56.961886 27040 leader_election.cc:288] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [CANDIDATE]: Term 2 election: Requested vote from peers 9f97442df1a34ff6a8a6215460b56d62 (127.15.254.62:41399), 5d972ddbeae549fba9841504b5db758a (127.15.254.61:36519)
I20260807 06:22:56.961912 27039 leader_election.cc:288] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [CANDIDATE]: Term 2 election: Requested vote from peers 9f97442df1a34ff6a8a6215460b56d62 (127.15.254.62:41399), 8e73d6f7d63a48379cbc6aa648137096 (127.15.254.60:44007)
I20260807 06:22:56.962409 27043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:56.962151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.0.0.1:48560)
0807 06:22:56.962255 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:56.962308 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260807 06:22:56.962473 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:56.962232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.0.0.1:48566)
0807 06:22:56.962326 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:56.962387 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260807 06:22:56.962611 26308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d972ddbeae549fba9841504b5db758a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8e73d6f7d63a48379cbc6aa648137096"
I20260807 06:22:56.962594 26045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8e73d6f7d63a48379cbc6aa648137096" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5d972ddbeae549fba9841504b5db758a"
I20260807 06:22:56.962779 26308 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5d972ddbeae549fba9841504b5db758a in current term 2: Already voted for candidate 8e73d6f7d63a48379cbc6aa648137096 in this term.
I20260807 06:22:56.962781 26045 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e73d6f7d63a48379cbc6aa648137096 in current term 2: Already voted for candidate 5d972ddbeae549fba9841504b5db758a in this term.
I20260807 06:22:56.963394 27043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:56.963025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.0.0.1:48580)
0807 06:22:56.963295 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:56.963340 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
W20260807 06:22:56.963503 26275 leader_election.cc:334] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9f97442df1a34ff6a8a6215460b56d62 (127.15.254.62:41399): Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
I20260807 06:22:56.963565 26275 leader_election.cc:302] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e73d6f7d63a48379cbc6aa648137096; no voters: 5d972ddbeae549fba9841504b5db758a, 9f97442df1a34ff6a8a6215460b56d62
I20260807 06:22:56.963660 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:56.963406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.0.0.1:48586)
0807 06:22:56.963524 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:56.963576 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260807 06:22:56.963711 27040 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260807 06:22:56.963814 26020 leader_election.cc:334] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9f97442df1a34ff6a8a6215460b56d62 (127.15.254.62:41399): Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
I20260807 06:22:56.963886 26020 leader_election.cc:302] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d972ddbeae549fba9841504b5db758a; no voters: 8e73d6f7d63a48379cbc6aa648137096, 9f97442df1a34ff6a8a6215460b56d62
I20260807 06:22:56.964039 27039 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 06:22:57.440037 27054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:57.439511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.3:57701)
0807 06:22:57.439816 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:57.439908 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":123,"threads_started":1}
I20260807 06:22:57.441408 27055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:57.440897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.1:38413)
0807 06:22:57.441150 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:57.441292 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":108,"threads_started":1}
I20260807 06:22:57.442793 27059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:57.442358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.2:35735)
0807 06:22:57.442618 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:57.442689 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
I20260807 06:22:58.441406 27060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:58.440888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.3:55291)
0807 06:22:58.441184 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:58.441261 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":85,"threads_started":1}
I20260807 06:22:58.442607 27061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:58.442170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.1:36905)
0807 06:22:58.442421 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:58.442487 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":66,"threads_started":1}
I20260807 06:22:58.444239 27062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:58.443711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.2:55647)
0807 06:22:58.444080 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:58.444143 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":99,"threads_started":1}
I20260807 06:22:58.799856 27063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:22:58.800089 27063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: false } }
I20260807 06:22:58.800655 27063 leader_election.cc:288] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9f97442df1a34ff6a8a6215460b56d62 (127.15.254.62:41399), 5d972ddbeae549fba9841504b5db758a (127.15.254.61:36519)
I20260807 06:22:58.801257 27064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:58.800800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.0.0.1:48852)
0807 06:22:58.801071 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:58.801156 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":78,"threads_started":1}
I20260807 06:22:58.801249 26045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8e73d6f7d63a48379cbc6aa648137096" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5d972ddbeae549fba9841504b5db758a" is_pre_election: true
I20260807 06:22:58.801448 26045 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e73d6f7d63a48379cbc6aa648137096 in term 2.
I20260807 06:22:58.801826 26270 leader_election.cc:302] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d972ddbeae549fba9841504b5db758a, 8e73d6f7d63a48379cbc6aa648137096; no voters:
I20260807 06:22:58.802018 27063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 06:22:58.802088 27063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:22:58.802134 27063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 2 FOLLOWER]: Advancing to term 3
I20260807 06:22:58.802265 27064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:58.801993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.0.0.1:48866)
0807 06:22:58.802128 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:58.802193 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260807 06:22:58.802443 26275 leader_election.cc:334] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9f97442df1a34ff6a8a6215460b56d62 (127.15.254.62:41399): Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
I20260807 06:22:58.803354 27063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: false } }
I20260807 06:22:58.803757 27063 leader_election.cc:288] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [CANDIDATE]: Term 3 election: Requested vote from peers 9f97442df1a34ff6a8a6215460b56d62 (127.15.254.62:41399), 5d972ddbeae549fba9841504b5db758a (127.15.254.61:36519)
I20260807 06:22:58.804090 27064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:58.803831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.0.0.1:48878)
0807 06:22:58.803934 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:58.804013 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260807 06:22:58.804189 26045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8e73d6f7d63a48379cbc6aa648137096" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5d972ddbeae549fba9841504b5db758a"
I20260807 06:22:58.804344 26045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 2 FOLLOWER]: Advancing to term 3
I20260807 06:22:58.804850 27064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:58.804633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.0.0.1:48882)
0807 06:22:58.804748 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:58.804798 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260807 06:22:58.804970 26275 leader_election.cc:334] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9f97442df1a34ff6a8a6215460b56d62 (127.15.254.62:41399): Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
I20260807 06:22:58.805579 26045 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e73d6f7d63a48379cbc6aa648137096 in term 3.
I20260807 06:22:58.805893 26270 leader_election.cc:302] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d972ddbeae549fba9841504b5db758a, 8e73d6f7d63a48379cbc6aa648137096; no voters: 9f97442df1a34ff6a8a6215460b56d62
I20260807 06:22:58.806143 27063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 06:22:58.806378 27063 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [term 3 LEADER]: Becoming Leader. State: Replica: 8e73d6f7d63a48379cbc6aa648137096, State: Running, Role: LEADER
I20260807 06:22:58.806715 27063 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: false } }
I20260807 06:22:58.808393 27068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e73d6f7d63a48379cbc6aa648137096. Latest consensus state: current_term: 3 leader_uuid: "8e73d6f7d63a48379cbc6aa648137096" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: false } } }
I20260807 06:22:58.808563 27068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:58.808926 27071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:22:58.811185 27071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:22:58.811604 27071 catalog_manager.cc:1259] Loaded cluster ID: b67ecb0da3614575836bf6cf3f400654
I20260807 06:22:58.811676 27071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:22:58.812237 27071 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:22:58.812724 27071 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096: Loaded TSK: 0
I20260807 06:22:58.813406 27071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:22:58.899480 26291 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:22:58.902873 26291 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:22:58.906389 27064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:58.906111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.0.0.1:48916)
0807 06:22:58.906261 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:58.906314 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260807 06:22:58.906397 26045 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [term 3 FOLLOWER]: Refusing update from remote peer 8e73d6f7d63a48379cbc6aa648137096: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260807 06:22:58.906940 27068 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260807 06:22:58.907226 27064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:58.906952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.0.0.1:48926)
0807 06:22:58.907081 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:58.907138 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 06:22:58.907364 26275 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 -> Peer 9f97442df1a34ff6a8a6215460b56d62 (127.15.254.62:41399): Couldn't send request to peer 9f97442df1a34ff6a8a6215460b56d62. Status: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 06:22:58.908581 27073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e73d6f7d63a48379cbc6aa648137096. Latest consensus state: current_term: 3 leader_uuid: "8e73d6f7d63a48379cbc6aa648137096" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: false } } }
I20260807 06:22:58.908728 27073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:22:58.910030 27068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "8e73d6f7d63a48379cbc6aa648137096" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: false } } }
I20260807 06:22:58.910218 27068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096 [sys.catalog]: This master's current role is: LEADER
I20260807 06:22:58.910360 27073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "8e73d6f7d63a48379cbc6aa648137096" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f97442df1a34ff6a8a6215460b56d62" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41399 } } peers { permanent_uuid: "5d972ddbeae549fba9841504b5db758a" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 36519 } attrs { promote: false } } peers { permanent_uuid: "8e73d6f7d63a48379cbc6aa648137096" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 44007 } attrs { promote: false } } }
I20260807 06:22:58.910473 27073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d972ddbeae549fba9841504b5db758a [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:22:58.921993 27078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:58.917075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:43515 (local address 127.0.0.1:42092)
0807 06:22:58.917470 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:58.917511 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:58.917734 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:58.918527 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:58.918531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:58.918554 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:58.918687 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:58.918694 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:58.919745 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:58.919748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:58.920847 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:58.920856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:58.921539 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:58.921543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:58.921636 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:58.921662 (+ 26us) client_negotiation.cc:769] Sending connection context
0807 06:22:58.921734 (+ 72us) client_negotiation.cc:240] Negotiation successful
0807 06:22:58.921809 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":188,"threads_started":1}
I20260807 06:22:58.922416 27081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:58.917465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42092 (local address 127.15.254.3:43515)
0807 06:22:58.918172 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:58.918176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:58.918189 (+ 13us) server_negotiation.cc:407] Connection header received
0807 06:22:58.918237 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:58.918239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:58.918288 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:58.918394 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:58.918784 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:58.919622 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:58.920971 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:58.921414 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:58.921537 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:58.921607 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:58.922172 (+ 565us) server_negotiation.cc:299] Negotiation successful
0807 06:22:58.922259 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":90,"threads_started":1}
I20260807 06:22:58.923175 26845 tablet_service.cc:1467] Processing CreateTablet for tablet 4df1c3f81f7749d5904f2f255e6940f6 (DEFAULT_TABLE table=test-table [id=8adb85b9b9b6434ca757bb836021fd7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:58.923267 27084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:58.918202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:42279 (local address 127.0.0.1:39162)
0807 06:22:58.918524 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:58.918537 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:58.918620 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:58.919219 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:58.919222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:58.919242 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:58.919345 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:58.919350 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:58.920376 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:58.920380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:58.921625 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:58.921636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:58.922816 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:58.922820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:58.922942 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:58.922971 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 06:22:58.923035 (+ 64us) client_negotiation.cc:240] Negotiation successful
0807 06:22:58.923104 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"spinlock_wait_cycles":384,"thread_start_us":116,"threads_started":1}
I20260807 06:22:58.923282 27064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:58.916719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:34939 (local address 127.0.0.1:55260)
0807 06:22:58.916904 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:58.916920 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:58.917794 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:58.919751 (+ 1957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:58.919755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:58.919771 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:58.919890 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:58.919905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:58.920989 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:58.920993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:58.922061 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:58.922072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:58.922979 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:58.922983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:58.923072 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:58.923091 (+ 19us) client_negotiation.cc:769] Sending connection context
0807 06:22:58.923134 (+ 43us) client_negotiation.cc:240] Negotiation successful
0807 06:22:58.923189 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":1}
I20260807 06:22:58.923552 26845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4df1c3f81f7749d5904f2f255e6940f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:58.923741 27085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:58.918653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39162 (local address 127.15.254.1:42279)
0807 06:22:58.918933 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:58.918938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:58.918954 (+ 16us) server_negotiation.cc:407] Connection header received
0807 06:22:58.919017 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:58.919020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:58.919072 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:58.919170 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:58.919484 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:58.920278 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:58.921788 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:58.922578 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:58.922676 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:58.922749 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:58.923530 (+ 781us) server_negotiation.cc:299] Negotiation successful
0807 06:22:58.923613 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":77,"threads_started":1}
I20260807 06:22:58.923768 27079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:58.917236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55260 (local address 127.15.254.2:34939)
0807 06:22:58.919521 (+ 2285us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:58.919526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:58.919541 (+ 15us) server_negotiation.cc:407] Connection header received
0807 06:22:58.919597 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:58.919600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:58.919650 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:58.919740 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:58.920031 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:58.920856 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:58.922211 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:58.922836 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:58.923469 (+ 633us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:58.923543 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:58.923576 (+ 33us) server_negotiation.cc:299] Negotiation successful
0807 06:22:58.923642 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2195,"thread_start_us":82,"threads_started":1}
I20260807 06:22:58.924256 26652 tablet_service.cc:1467] Processing CreateTablet for tablet 4df1c3f81f7749d5904f2f255e6940f6 (DEFAULT_TABLE table=test-table [id=8adb85b9b9b6434ca757bb836021fd7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:58.924276 26454 tablet_service.cc:1467] Processing CreateTablet for tablet 4df1c3f81f7749d5904f2f255e6940f6 (DEFAULT_TABLE table=test-table [id=8adb85b9b9b6434ca757bb836021fd7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:22:58.924540 26652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4df1c3f81f7749d5904f2f255e6940f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:58.924566 26454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4df1c3f81f7749d5904f2f255e6940f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:22:58.926563 27087 tablet_bootstrap.cc:492] T 4df1c3f81f7749d5904f2f255e6940f6 P 043fb1b788ed4af18200209e0ac9f5c0: Bootstrap starting.
I20260807 06:22:58.927529 27092 tablet_bootstrap.cc:492] T 4df1c3f81f7749d5904f2f255e6940f6 P dc9fbde76a594443bf8b835f3b566d63: Bootstrap starting.
I20260807 06:22:58.927590 27087 tablet_bootstrap.cc:654] T 4df1c3f81f7749d5904f2f255e6940f6 P 043fb1b788ed4af18200209e0ac9f5c0: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:58.928347 27090 tablet_bootstrap.cc:492] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d: Bootstrap starting.
I20260807 06:22:58.928694 27092 tablet_bootstrap.cc:654] T 4df1c3f81f7749d5904f2f255e6940f6 P dc9fbde76a594443bf8b835f3b566d63: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:58.928951 27087 tablet_bootstrap.cc:492] T 4df1c3f81f7749d5904f2f255e6940f6 P 043fb1b788ed4af18200209e0ac9f5c0: No bootstrap required, opened a new log
I20260807 06:22:58.929055 27087 ts_tablet_manager.cc:1397] T 4df1c3f81f7749d5904f2f255e6940f6 P 043fb1b788ed4af18200209e0ac9f5c0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 06:22:58.929426 27087 raft_consensus.cc:348] T 4df1c3f81f7749d5904f2f255e6940f6 P 043fb1b788ed4af18200209e0ac9f5c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34939 } } peers { permanent_uuid: "043fb1b788ed4af18200209e0ac9f5c0" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43515 } } peers { permanent_uuid: "dc9fbde76a594443bf8b835f3b566d63" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42279 } }
I20260807 06:22:58.929545 27087 raft_consensus.cc:374] T 4df1c3f81f7749d5904f2f255e6940f6 P 043fb1b788ed4af18200209e0ac9f5c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:58.929586 27087 raft_consensus.cc:729] T 4df1c3f81f7749d5904f2f255e6940f6 P 043fb1b788ed4af18200209e0ac9f5c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 043fb1b788ed4af18200209e0ac9f5c0, State: Initialized, Role: FOLLOWER
I20260807 06:22:58.929711 27087 consensus_queue.cc:260] T 4df1c3f81f7749d5904f2f255e6940f6 P 043fb1b788ed4af18200209e0ac9f5c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34939 } } peers { permanent_uuid: "043fb1b788ed4af18200209e0ac9f5c0" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43515 } } peers { permanent_uuid: "dc9fbde76a594443bf8b835f3b566d63" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42279 } }
I20260807 06:22:58.930128 27087 ts_tablet_manager.cc:1428] T 4df1c3f81f7749d5904f2f255e6940f6 P 043fb1b788ed4af18200209e0ac9f5c0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 06:22:58.930207 27092 tablet_bootstrap.cc:492] T 4df1c3f81f7749d5904f2f255e6940f6 P dc9fbde76a594443bf8b835f3b566d63: No bootstrap required, opened a new log
I20260807 06:22:58.930310 27092 ts_tablet_manager.cc:1397] T 4df1c3f81f7749d5904f2f255e6940f6 P dc9fbde76a594443bf8b835f3b566d63: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 06:22:58.930733 27060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:58.930420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.3:55245)
0807 06:22:58.930587 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:58.930655 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260807 06:22:58.930800 27092 raft_consensus.cc:348] T 4df1c3f81f7749d5904f2f255e6940f6 P dc9fbde76a594443bf8b835f3b566d63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34939 } } peers { permanent_uuid: "043fb1b788ed4af18200209e0ac9f5c0" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43515 } } peers { permanent_uuid: "dc9fbde76a594443bf8b835f3b566d63" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42279 } }
I20260807 06:22:58.930914 27092 raft_consensus.cc:374] T 4df1c3f81f7749d5904f2f255e6940f6 P dc9fbde76a594443bf8b835f3b566d63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:58.930951 27092 raft_consensus.cc:729] T 4df1c3f81f7749d5904f2f255e6940f6 P dc9fbde76a594443bf8b835f3b566d63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9fbde76a594443bf8b835f3b566d63, State: Initialized, Role: FOLLOWER
I20260807 06:22:58.931155 27092 consensus_queue.cc:260] T 4df1c3f81f7749d5904f2f255e6940f6 P dc9fbde76a594443bf8b835f3b566d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34939 } } peers { permanent_uuid: "043fb1b788ed4af18200209e0ac9f5c0" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43515 } } peers { permanent_uuid: "dc9fbde76a594443bf8b835f3b566d63" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42279 } }
I20260807 06:22:58.931516 27090 tablet_bootstrap.cc:654] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d: Neither blocks nor log segments found. Creating new log.
I20260807 06:22:58.931540 27092 ts_tablet_manager.cc:1428] T 4df1c3f81f7749d5904f2f255e6940f6 P dc9fbde76a594443bf8b835f3b566d63: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 06:22:58.932891 27090 tablet_bootstrap.cc:492] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d: No bootstrap required, opened a new log
I20260807 06:22:58.932977 27090 ts_tablet_manager.cc:1397] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260807 06:22:58.933457 27061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:58.933148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.1:51257)
0807 06:22:58.933318 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:58.933379 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260807 06:22:58.933440 27090 raft_consensus.cc:348] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34939 } } peers { permanent_uuid: "043fb1b788ed4af18200209e0ac9f5c0" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43515 } } peers { permanent_uuid: "dc9fbde76a594443bf8b835f3b566d63" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42279 } }
I20260807 06:22:58.933558 27090 raft_consensus.cc:374] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:22:58.933593 27090 raft_consensus.cc:729] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e00ec4b6044d4ea221ea3da197ad7d, State: Initialized, Role: FOLLOWER
I20260807 06:22:58.933712 27090 consensus_queue.cc:260] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34939 } } peers { permanent_uuid: "043fb1b788ed4af18200209e0ac9f5c0" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43515 } } peers { permanent_uuid: "dc9fbde76a594443bf8b835f3b566d63" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42279 } }
I20260807 06:22:58.934255 27090 ts_tablet_manager.cc:1428] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260807 06:22:58.935223 27062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:58.934938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.2:54743)
0807 06:22:58.935067 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:58.935137 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260807 06:22:58.936125 26939 heartbeater.cc:503] Master 127.15.254.60:44007 was elected leader, sending a full tablet report...
I20260807 06:22:58.936589 27062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:58.936286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.2:59135)
0807 06:22:58.936415 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:58.936503 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260807 06:22:58.936980 27060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:58.936717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.3:36781)
0807 06:22:58.936831 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:58.936894 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260807 06:22:58.938046 26549 heartbeater.cc:503] Master 127.15.254.60:44007 was elected leader, sending a full tablet report...
I20260807 06:22:58.938563 27061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:58.938242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.1:52537)
0807 06:22:58.938437 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:58.938491 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260807 06:22:59.276171 27101 raft_consensus.cc:482] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:22:59.276353 27101 raft_consensus.cc:504] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34939 } } peers { permanent_uuid: "043fb1b788ed4af18200209e0ac9f5c0" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43515 } } peers { permanent_uuid: "dc9fbde76a594443bf8b835f3b566d63" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42279 } }
I20260807 06:22:59.277782 27101 leader_election.cc:288] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 043fb1b788ed4af18200209e0ac9f5c0 (127.15.254.3:43515), dc9fbde76a594443bf8b835f3b566d63 (127.15.254.1:42279)
I20260807 06:22:59.281780 27081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:59.278197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:56625 (local address 127.15.254.3:43515)
0807 06:22:59.278362 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:59.278367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:59.278384 (+ 17us) server_negotiation.cc:407] Connection header received
0807 06:22:59.278597 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:59.278599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:59.278646 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:59.278735 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:59.279082 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:59.279605 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:59.280758 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:59.281163 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:59.281301 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:59.281368 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:59.281581 (+ 213us) server_negotiation.cc:299] Negotiation successful
0807 06:22:59.281654 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 06:22:59.281780 27062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:59.278003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:43515 (local address 127.15.254.2:56625)
0807 06:22:59.278250 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:59.278270 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:59.278429 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:59.278750 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:59.278753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:59.278786 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:59.278947 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:59.278953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:59.279720 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:59.279725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:59.280619 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:59.280629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:59.281324 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:59.281328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:59.281438 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:59.281458 (+ 20us) client_negotiation.cc:769] Sending connection context
0807 06:22:59.281551 (+ 93us) client_negotiation.cc:240] Negotiation successful
0807 06:22:59.281629 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":3,"spinlock_wait_cycles":6016}
I20260807 06:22:59.282073 27102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:59.278188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:42279 (local address 127.15.254.2:45051)
0807 06:22:59.278625 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:59.278636 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 06:22:59.278787 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:22:59.279102 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:22:59.279105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:22:59.279122 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:22:59.279243 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:59.279249 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:59.280014 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:59.280019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:59.280923 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:22:59.280933 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:22:59.281599 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:22:59.281603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:22:59.281717 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:59.281740 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 06:22:59.281829 (+ 89us) client_negotiation.cc:240] Negotiation successful
0807 06:22:59.281911 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":18,"thread_start_us":178,"threads_started":1}
I20260807 06:22:59.282056 27085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:59.278302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:45051 (local address 127.15.254.1:42279)
0807 06:22:59.278446 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:59.278451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:59.278740 (+ 289us) server_negotiation.cc:407] Connection header received
0807 06:22:59.278921 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:59.278925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:59.278991 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:59.279088 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:22:59.279363 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:59.279892 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:59.281054 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:59.281471 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:59.281608 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:59.281665 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:59.281850 (+ 185us) server_negotiation.cc:299] Negotiation successful
0807 06:22:59.281928 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 06:22:59.282411 26871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df1c3f81f7749d5904f2f255e6940f6" candidate_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "043fb1b788ed4af18200209e0ac9f5c0" is_pre_election: true
I20260807 06:22:59.282591 26871 raft_consensus.cc:2454] T 4df1c3f81f7749d5904f2f255e6940f6 P 043fb1b788ed4af18200209e0ac9f5c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e00ec4b6044d4ea221ea3da197ad7d in term 0.
I20260807 06:22:59.282790 26487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df1c3f81f7749d5904f2f255e6940f6" candidate_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fbde76a594443bf8b835f3b566d63" is_pre_election: true
I20260807 06:22:59.283078 26487 raft_consensus.cc:2454] T 4df1c3f81f7749d5904f2f255e6940f6 P dc9fbde76a594443bf8b835f3b566d63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e00ec4b6044d4ea221ea3da197ad7d in term 0.
I20260807 06:22:59.283098 26587 leader_election.cc:302] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 043fb1b788ed4af18200209e0ac9f5c0, f0e00ec4b6044d4ea221ea3da197ad7d; no voters:
I20260807 06:22:59.283408 27101 raft_consensus.cc:2781] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:22:59.283504 27101 raft_consensus.cc:482] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:22:59.283567 27101 raft_consensus.cc:3037] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:59.285113 27101 raft_consensus.cc:504] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34939 } } peers { permanent_uuid: "043fb1b788ed4af18200209e0ac9f5c0" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43515 } } peers { permanent_uuid: "dc9fbde76a594443bf8b835f3b566d63" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42279 } }
I20260807 06:22:59.285565 27101 leader_election.cc:288] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d [CANDIDATE]: Term 1 election: Requested vote from peers 043fb1b788ed4af18200209e0ac9f5c0 (127.15.254.3:43515), dc9fbde76a594443bf8b835f3b566d63 (127.15.254.1:42279)
I20260807 06:22:59.285991 26487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df1c3f81f7749d5904f2f255e6940f6" candidate_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fbde76a594443bf8b835f3b566d63"
I20260807 06:22:59.285991 26871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df1c3f81f7749d5904f2f255e6940f6" candidate_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "043fb1b788ed4af18200209e0ac9f5c0"
I20260807 06:22:59.286136 26871 raft_consensus.cc:3037] T 4df1c3f81f7749d5904f2f255e6940f6 P 043fb1b788ed4af18200209e0ac9f5c0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:59.286136 26487 raft_consensus.cc:3037] T 4df1c3f81f7749d5904f2f255e6940f6 P dc9fbde76a594443bf8b835f3b566d63 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:22:59.287503 26487 raft_consensus.cc:2454] T 4df1c3f81f7749d5904f2f255e6940f6 P dc9fbde76a594443bf8b835f3b566d63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e00ec4b6044d4ea221ea3da197ad7d in term 1.
I20260807 06:22:59.287503 26871 raft_consensus.cc:2454] T 4df1c3f81f7749d5904f2f255e6940f6 P 043fb1b788ed4af18200209e0ac9f5c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e00ec4b6044d4ea221ea3da197ad7d in term 1.
I20260807 06:22:59.287837 26583 leader_election.cc:302] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc9fbde76a594443bf8b835f3b566d63, f0e00ec4b6044d4ea221ea3da197ad7d; no voters:
I20260807 06:22:59.288108 27101 raft_consensus.cc:2781] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:22:59.288421 27101 raft_consensus.cc:686] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d [term 1 LEADER]: Becoming Leader. State: Replica: f0e00ec4b6044d4ea221ea3da197ad7d, State: Running, Role: LEADER
I20260807 06:22:59.288820 26741 heartbeater.cc:503] Master 127.15.254.60:44007 was elected leader, sending a full tablet report...
I20260807 06:22:59.288782 27101 consensus_queue.cc:237] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34939 } } peers { permanent_uuid: "043fb1b788ed4af18200209e0ac9f5c0" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43515 } } peers { permanent_uuid: "dc9fbde76a594443bf8b835f3b566d63" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42279 } }
I20260807 06:22:59.289367 27102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:59.289093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.2:49993)
0807 06:22:59.289223 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:59.289285 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260807 06:22:59.289512 27084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:59.289240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.0.0.1:48932)
0807 06:22:59.289353 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:59.289412 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260807 06:22:59.290381 27084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:59.290078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.0.0.1:48940)
0807 06:22:59.290225 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:59.290280 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260807 06:22:59.291543 26293 catalog_manager.cc:5495] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e00ec4b6044d4ea221ea3da197ad7d (127.15.254.2). New cstate: current_term: 1 leader_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e00ec4b6044d4ea221ea3da197ad7d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "043fb1b788ed4af18200209e0ac9f5c0" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fbde76a594443bf8b835f3b566d63" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42279 } health_report { overall_health: UNKNOWN } } }
I20260807 06:22:59.386518 27079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:59.382670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.15.254.2:34939)
0807 06:22:59.382822 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0807 06:22:59.382827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:22:59.382845 (+ 18us) server_negotiation.cc:407] Connection header received
0807 06:22:59.382987 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:22:59.382990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:22:59.383050 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:22:59.383174 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:22:59.383428 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:59.383997 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:59.385031 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:22:59.385255 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:22:59.385367 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:22:59.385660 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:22:59.386103 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:22:59.386237 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0807 06:22:59.386305 (+ 68us) server_negotiation.cc:299] Negotiation successful
0807 06:22:59.386363 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 06:22:59.389597 26871 raft_consensus.cc:1261] T 4df1c3f81f7749d5904f2f255e6940f6 P 043fb1b788ed4af18200209e0ac9f5c0 [term 1 FOLLOWER]: Refusing update from remote peer f0e00ec4b6044d4ea221ea3da197ad7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:22:59.389858 26487 raft_consensus.cc:1261] T 4df1c3f81f7749d5904f2f255e6940f6 P dc9fbde76a594443bf8b835f3b566d63 [term 1 FOLLOWER]: Refusing update from remote peer f0e00ec4b6044d4ea221ea3da197ad7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:22:59.390208 27103 consensus_queue.cc:1035] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "043fb1b788ed4af18200209e0ac9f5c0" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:22:59.390547 27101 consensus_queue.cc:1035] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fbde76a594443bf8b835f3b566d63" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:22:59.392831 27060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:59.392561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.3:41831)
0807 06:22:59.392686 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:59.392751 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260807 06:22:59.393846 27061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:59.393343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.1:43845)
0807 06:22:59.393689 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:59.393757 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263}
I20260807 06:22:59.396263 27102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:59.395652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.2:33341)
0807 06:22:59.396118 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:59.396183 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388}
I20260807 06:22:59.396417 27108 mvcc.cc:205] Tried to move back new op lower bound from 7315799160374235136 to 7315799159970160640. Current Snapshot: MvccSnapshot[applied={T|T < 7315799160374235136}]
I20260807 06:22:59.396801 27107 mvcc.cc:205] Tried to move back new op lower bound from 7315799160374235136 to 7315799159970160640. Current Snapshot: MvccSnapshot[applied={T|T < 7315799160374235136 or (T in {7315799160374235136})}]
I20260807 06:22:59.398453 27060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:59.396266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.3:57767)
0807 06:22:59.398305 (+ 2039us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:59.398385 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1957}
I20260807 06:22:59.400825 27102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:59.400542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.2:52965)
0807 06:22:59.400683 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:59.400738 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260807 06:22:59.403993 27061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:22:59.403315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41399 (local address 127.15.254.1:39333)
0807 06:22:59.403846 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:59.403911 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448}
I20260807 06:22:59.404320 27112 mvcc.cc:205] Tried to move back new op lower bound from 7315799160374235136 to 7315799159970160640. Current Snapshot: MvccSnapshot[applied={T|T < 7315799160374235136}]
I20260807 06:22:59.404572 26291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:22:59.404728 26291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:22:59.409343 26291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e73d6f7d63a48379cbc6aa648137096: Sending DeleteTablet for 3 replicas of tablet 4df1c3f81f7749d5904f2f255e6940f6
I20260807 06:22:59.410077 26845 tablet_service.cc:1514] Processing DeleteTablet for tablet 4df1c3f81f7749d5904f2f255e6940f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:59 UTC) from {username='oryx', principal='oryx/127.15.254.60@KRBTEST.COM'} at 127.0.0.1:42092
I20260807 06:22:59.410429 26652 tablet_service.cc:1514] Processing DeleteTablet for tablet 4df1c3f81f7749d5904f2f255e6940f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:59 UTC) from {username='oryx', principal='oryx/127.15.254.60@KRBTEST.COM'} at 127.0.0.1:55260
I20260807 06:22:59.410477 27122 tablet_replica.cc:330] T 4df1c3f81f7749d5904f2f255e6940f6 P 043fb1b788ed4af18200209e0ac9f5c0: stopping tablet replica
I20260807 06:22:59.410564 26454 tablet_service.cc:1514] Processing DeleteTablet for tablet 4df1c3f81f7749d5904f2f255e6940f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:22:59 UTC) from {username='oryx', principal='oryx/127.15.254.60@KRBTEST.COM'} at 127.0.0.1:39162
I20260807 06:22:59.410611 27122 raft_consensus.cc:2229] T 4df1c3f81f7749d5904f2f255e6940f6 P 043fb1b788ed4af18200209e0ac9f5c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:22:59.410742 27122 raft_consensus.cc:2258] T 4df1c3f81f7749d5904f2f255e6940f6 P 043fb1b788ed4af18200209e0ac9f5c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:59.411248 27124 tablet_replica.cc:330] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d: stopping tablet replica
I20260807 06:22:59.411419 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 26240
I20260807 06:22:59.414042 27122 ts_tablet_manager.cc:1905] T 4df1c3f81f7749d5904f2f255e6940f6 P 043fb1b788ed4af18200209e0ac9f5c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:22:59.417984 27122 ts_tablet_manager.cc:1918] T 4df1c3f81f7749d5904f2f255e6940f6 P 043fb1b788ed4af18200209e0ac9f5c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:22:59.418177 27122 log.cc:1196] T 4df1c3f81f7749d5904f2f255e6940f6 P 043fb1b788ed4af18200209e0ac9f5c0: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/4df1c3f81f7749d5904f2f255e6940f6
I20260807 06:22:59.428977 27122 ts_tablet_manager.cc:1939] T 4df1c3f81f7749d5904f2f255e6940f6 P 043fb1b788ed4af18200209e0ac9f5c0: Deleting consensus metadata
I20260807 06:22:59.430753 26275 catalog_manager.cc:4841] TS 043fb1b788ed4af18200209e0ac9f5c0 (127.15.254.3:43515): tablet 4df1c3f81f7749d5904f2f255e6940f6 (table test-table [id=8adb85b9b9b6434ca757bb836021fd7e]) successfully deleted
I20260807 06:22:59.411455 27124 raft_consensus.cc:2229] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:22:59.431571 27124 raft_consensus.cc:2258] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:22:59.433650 27124 ts_tablet_manager.cc:1905] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:22:59.436851 27124 ts_tablet_manager.cc:1918] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:22:59.437029 27124 log.cc:1196] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/4df1c3f81f7749d5904f2f255e6940f6
I20260807 06:22:59.437434 27124 ts_tablet_manager.cc:1939] T 4df1c3f81f7749d5904f2f255e6940f6 P f0e00ec4b6044d4ea221ea3da197ad7d: Deleting consensus metadata
I20260807 06:22:59.438875 26274 catalog_manager.cc:4841] TS f0e00ec4b6044d4ea221ea3da197ad7d (127.15.254.2:34939): tablet 4df1c3f81f7749d5904f2f255e6940f6 (table test-table [id=8adb85b9b9b6434ca757bb836021fd7e]) successfully deleted
I20260807 06:22:59.447335 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 26562
W20260807 06:22:59.447371 27084 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:22:59.447015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:42279 (local address 127.0.0.1:39176)
0807 06:22:59.447199 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0807 06:22:59.447259 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.1:42279: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":74}
W20260807 06:22:59.450053 26270 catalog_manager.cc:4584] TS dc9fbde76a594443bf8b835f3b566d63 (127.15.254.1:42279): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4df1c3f81f7749d5904f2f255e6940f6: Network error: Client connection negotiation failed: client connection to 127.15.254.1:42279: connect: Connection reset by peer (error 104)
I20260807 06:22:59.458211 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 26751
I20260807 06:22:59.473618 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 25848
I20260807 06:22:59.489773 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 26002
2026-08-07T06:22:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8139 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:22:59 dist-test-slave-44d2 krb5kdc[27133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:22:59 dist-test-slave-44d2 krb5kdc[27133](info): set up 2 sockets
Aug 07 06:22:59 dist-test-slave-44d2 krb5kdc[27133](info): commencing operation
krb5kdc: starting...
W20260807 06:23:00.747545 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.224s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:23:00 dist-test-slave-44d2 krb5kdc[27133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:23:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:23:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.62.keytab.
Entry for principal oryx/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.62.keytab.
I20260807 06:23:00.891718 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:33815
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.62.keytab
--principal=oryx/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:34831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:33815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:23:01.013980 27152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:23:01.014308 27152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:23:01.014350 27152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:23:01.018486 27152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:23:01.018553 27152 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 06:23:01.018570 27152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:23:01.018589 27152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:23:01.018605 27152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:23:01.018657 27152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:23:01.024272 27152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:34831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:33815
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.62.keytab
--principal=oryx/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:33815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:23:01.025533 27152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:23:01.026650 27152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 06:23:01.036443 27152 server_base.cc:1034] running on GCE node
W20260807 06:23:01.036384 27159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:01.036391 27158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:01.036449 27161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:01.037429 27152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:23:01.038797 27152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:23:01.040071 27152 hybrid_clock.cc:648] HybridClock initialized: now 1786083781040026 us; error 60 us; skew 500 ppm
Aug 07 06:23:01 dist-test-slave-44d2 krb5kdc[27133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083781, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:23:01.043561 27152 init.cc:377] Logged in from keytab as oryx/127.15.254.62@KRBTEST.COM (short username oryx)
I20260807 06:23:01.044718 27152 webserver.cc:460] Webserver started at http://127.15.254.62:35441/ using document root <none> and password file <none>
I20260807 06:23:01.045332 27152 fs_manager.cc:359] Metadata directory not provided
I20260807 06:23:01.045413 27152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:23:01.045648 27152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:23:01.047907 27152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "ff5b7a8e83ca458c896ba0d82513a6e7"
format_stamp: "Formatted at 2026-08-07 06:23:01 on dist-test-slave-44d2"
server_key: "25e5aa8707f61a7d31acafcdeb56e07b"
server_key_iv: "3a30bda5f02b56e75955f4be84685434"
server_key_version: "encryptionkey@0"
I20260807 06:23:01.048565 27152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "ff5b7a8e83ca458c896ba0d82513a6e7"
format_stamp: "Formatted at 2026-08-07 06:23:01 on dist-test-slave-44d2"
server_key: "25e5aa8707f61a7d31acafcdeb56e07b"
server_key_iv: "3a30bda5f02b56e75955f4be84685434"
server_key_version: "encryptionkey@0"
I20260807 06:23:01.052754 27152 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260807 06:23:01.055344 27172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:01.056768 27152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 06:23:01.056883 27152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "ff5b7a8e83ca458c896ba0d82513a6e7"
format_stamp: "Formatted at 2026-08-07 06:23:01 on dist-test-slave-44d2"
server_key: "25e5aa8707f61a7d31acafcdeb56e07b"
server_key_iv: "3a30bda5f02b56e75955f4be84685434"
server_key_version: "encryptionkey@0"
I20260807 06:23:01.056977 27152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:23:01.077557 27152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:23:01.081562 27152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:23:01.081761 27152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:23:01.090955 27152 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:33815
I20260807 06:23:01.090987 27260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:33815 every 8 connection(s)
I20260807 06:23:01.092375 27152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:23:01.095382 27261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:23:01.100447 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 27152
I20260807 06:23:01.100625 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:23:01.100960 16376 external_mini_cluster.cc:1428] Setting key 0fcf80ad2ddc30571b8685e7c17cca51
I20260807 06:23:01.101611 27261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7: Bootstrap starting.
I20260807 06:23:01.104182 27261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7: Neither blocks nor log segments found. Creating new log.
I20260807 06:23:01.105027 27261 log.cc:824] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7: Log is configured to *not* fsync() on all Append() calls
I20260807 06:23:01.106910 27261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7: No bootstrap required, opened a new log
Aug 07 06:23:01 dist-test-slave-44d2 krb5kdc[27133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM
I20260807 06:23:01.110451 27261 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5b7a8e83ca458c896ba0d82513a6e7" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 33815 } }
I20260807 06:23:01.110786 27261 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:23:01.110878 27261 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5b7a8e83ca458c896ba0d82513a6e7, State: Initialized, Role: FOLLOWER
I20260807 06:23:01.111346 27261 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5b7a8e83ca458c896ba0d82513a6e7" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 33815 } }
I20260807 06:23:01.111507 27261 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:23:01.111598 27261 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:23:01.111703 27261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:23:01.112972 27261 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5b7a8e83ca458c896ba0d82513a6e7" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 33815 } }
I20260807 06:23:01.113354 27261 leader_election.cc:302] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff5b7a8e83ca458c896ba0d82513a6e7; no voters:
I20260807 06:23:01.113729 27261 leader_election.cc:288] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:23:01.113847 27267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:23:01.114095 27267 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7 [term 1 LEADER]: Becoming Leader. State: Replica: ff5b7a8e83ca458c896ba0d82513a6e7, State: Running, Role: LEADER
I20260807 06:23:01.114521 27267 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5b7a8e83ca458c896ba0d82513a6e7" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 33815 } }
I20260807 06:23:01.115054 27261 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:23:01.116294 27268 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff5b7a8e83ca458c896ba0d82513a6e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5b7a8e83ca458c896ba0d82513a6e7" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 33815 } } }
I20260807 06:23:01.116541 27268 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7 [sys.catalog]: This master's current role is: LEADER
I20260807 06:23:01.116864 27268 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff5b7a8e83ca458c896ba0d82513a6e7. Latest consensus state: current_term: 1 leader_uuid: "ff5b7a8e83ca458c896ba0d82513a6e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5b7a8e83ca458c896ba0d82513a6e7" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 33815 } } }
I20260807 06:23:01.117007 27268 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7 [sys.catalog]: This master's current role is: LEADER
I20260807 06:23:01.117462 27265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:01.102495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59226 (local address 127.15.254.62:33815)
0807 06:23:01.103038 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:01.103056 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:01.103096 (+ 40us) server_negotiation.cc:407] Connection header received
0807 06:23:01.103790 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:01.103808 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:01.104175 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:01.104629 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:01.105581 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.106593 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.107571 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.108056 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.108261 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:01.111300 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:01.111306 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:01.111319 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:23:01.111351 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:01.113539 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:01.114226 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:01.114232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:01.114238 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:01.114384 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:01.114725 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:01.114728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:01.114730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:01.115253 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:01.115412 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:01.115908 (+ 496us) server_negotiation.cc:299] Negotiation successful
0807 06:23:01.116317 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":156,"threads_started":1}
W20260807 06:23:01.120842 27291 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:23:01.120947 27291 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:23:01.121057 27286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:23:01.121830 27286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:23:01.128907 27286 catalog_manager.cc:1347] Generated new cluster ID: cf891c3684ff4f679f3b28e554415bff
I20260807 06:23:01.129011 27286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:23:01.137077 27286 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:23:01.138007 27286 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:23:01.151072 27286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7: Generated new TSK 0
I20260807 06:23:01.151820 27286 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.1.keytab.
Entry for principal oryx/127.15.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.1.keytab.
I20260807 06:23:01.188148 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:0
--local_ip_for_outbound_sockets=127.15.254.1
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:33815
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.1.keytab
--principal=oryx/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:34831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:23:01.325939 27301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:23:01.326251 27301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:23:01.326297 27301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:23:01.330600 27301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:23:01.330665 27301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:23:01.330706 27301 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 06:23:01.330778 27301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:23:01.336256 27301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:34831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.1.keytab
--principal=oryx/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.254.62:33815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:23:01.337383 27301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:23:01.338443 27301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:23:01.348686 27307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:01.348752 27301 server_base.cc:1034] running on GCE node
W20260807 06:23:01.348698 27308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:01.348686 27314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:01.349481 27301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:23:01.350129 27301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:23:01.351306 27301 hybrid_clock.cc:648] HybridClock initialized: now 1786083781351287 us; error 32 us; skew 500 ppm
Aug 07 06:23:01 dist-test-slave-44d2 krb5kdc[27133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083781, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:23:01.354554 27301 init.cc:377] Logged in from keytab as oryx/127.15.254.1@KRBTEST.COM (short username oryx)
I20260807 06:23:01.355491 27301 webserver.cc:460] Webserver started at http://127.15.254.1:36839/ using document root <none> and password file <none>
I20260807 06:23:01.356091 27301 fs_manager.cc:359] Metadata directory not provided
I20260807 06:23:01.356154 27301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:23:01.356361 27301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:23:01.358795 27301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-0/data/instance:
uuid: "2d41a707b8bc4282ae3a5f954f607358"
format_stamp: "Formatted at 2026-08-07 06:23:01 on dist-test-slave-44d2"
server_key: "01c03eaf7f8d69866b34f73005bd41d3"
server_key_iv: "5fb926e157da82a6cb8bd5aea8ee5643"
server_key_version: "encryptionkey@0"
I20260807 06:23:01.359511 27301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance:
uuid: "2d41a707b8bc4282ae3a5f954f607358"
format_stamp: "Formatted at 2026-08-07 06:23:01 on dist-test-slave-44d2"
server_key: "01c03eaf7f8d69866b34f73005bd41d3"
server_key_iv: "5fb926e157da82a6cb8bd5aea8ee5643"
server_key_version: "encryptionkey@0"
I20260807 06:23:01.363977 27301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 06:23:01.366887 27327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:01.368242 27301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:23:01.368346 27301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "2d41a707b8bc4282ae3a5f954f607358"
format_stamp: "Formatted at 2026-08-07 06:23:01 on dist-test-slave-44d2"
server_key: "01c03eaf7f8d69866b34f73005bd41d3"
server_key_iv: "5fb926e157da82a6cb8bd5aea8ee5643"
server_key_version: "encryptionkey@0"
I20260807 06:23:01.368438 27301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:23:01.412056 27301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:23:01.415390 27301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:23:01.415553 27301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:23:01.417424 27301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:23:01.417479 27301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:01.417552 27301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:23:01.417649 27301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 07 06:23:01 dist-test-slave-44d2 krb5kdc[27133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083781, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM
I20260807 06:23:01.429090 27301 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:46267
I20260807 06:23:01.429181 27497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:46267 every 8 connection(s)
I20260807 06:23:01.430320 27301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:23:01.433534 27265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:01.418256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:58713 (local address 127.15.254.62:33815)
0807 06:23:01.418447 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:01.418454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:01.419466 (+ 1012us) server_negotiation.cc:407] Connection header received
0807 06:23:01.420840 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:01.420845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:01.420926 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:01.420993 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:01.423037 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.423928 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.425658 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.426053 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.426186 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:01.429326 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:01.429330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:01.429333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:23:01.429362 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:01.431255 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:01.431796 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:01.431800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:01.431802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:01.431903 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:01.432305 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:01.432309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:01.432311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:01.432559 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:01.432694 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:01.433218 (+ 524us) server_negotiation.cc:299] Negotiation successful
0807 06:23:01.433355 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 06:23:01.434324 27337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:01.418747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:33815 (local address 127.15.254.1:58713)
0807 06:23:01.419273 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:01.419329 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:01.420541 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:01.421515 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:01.421525 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:01.422062 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:01.422856 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:01.422881 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.424089 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.424094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:01.425505 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:01.425518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.426261 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.426267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:01.426494 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:01.427204 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:01.427222 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:01.429131 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:23:01.431380 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:23:01.431386 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:23:01.431400 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:23:01.431675 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:23:01.432003 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:23:01.432010 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:23:01.432014 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:23:01.432210 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:23:01.432694 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:23:01.432702 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:23:01.432956 (+ 254us) client_negotiation.cc:769] Sending connection context
0807 06:23:01.433155 (+ 199us) client_negotiation.cc:240] Negotiation successful
0807 06:23:01.433486 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":162,"threads_started":1}
I20260807 06:23:01.435688 27499 heartbeater.cc:347] Connected to a master server at 127.15.254.62:33815
I20260807 06:23:01.435869 27499 heartbeater.cc:464] Registering TS with master...
I20260807 06:23:01.436478 27499 heartbeater.cc:511] Master 127.15.254.62:33815 requested a full tablet report, sending...
I20260807 06:23:01.438024 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 27301
I20260807 06:23:01.438189 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance
I20260807 06:23:01.438496 27195 ts_manager.cc:194] Registered new tserver with Master: 2d41a707b8bc4282ae3a5f954f607358 (127.15.254.1:46267)
I20260807 06:23:01.438584 16376 external_mini_cluster.cc:1428] Setting key 2bea148555a743ac411edd1a2f976bf9
I20260807 06:23:01.439924 27195 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.254.1@KRBTEST.COM'} at 127.15.254.1:58713
I20260807 06:23:01.446702 27505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:01.438097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:33815 (local address 127.15.254.1:40197)
0807 06:23:01.438451 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:01.438467 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:01.438570 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:01.439457 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:01.439461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:01.439731 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:01.439914 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:01.439925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.441092 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.441096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:01.441693 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:01.441702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.442180 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.442183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:01.442284 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:01.442865 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:01.442877 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:01.443263 (+ 386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:23:01.445096 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:23:01.445100 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:23:01.445102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:23:01.445345 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:23:01.445653 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:23:01.445656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:23:01.445658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:23:01.445748 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:23:01.446131 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:23:01.446134 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:23:01.446235 (+ 101us) client_negotiation.cc:769] Sending connection context
0807 06:23:01.446346 (+ 111us) client_negotiation.cc:240] Negotiation successful
0807 06:23:01.446501 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":134,"threads_started":1}
I20260807 06:23:01.446693 27265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:01.439046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:40197 (local address 127.15.254.62:33815)
0807 06:23:01.439195 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:01.439200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:01.439216 (+ 16us) server_negotiation.cc:407] Connection header received
0807 06:23:01.439256 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:01.439258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:01.439300 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:01.439388 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:01.440164 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.440918 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.441834 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.442060 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.442171 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:01.443407 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:01.443411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:01.443414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:23:01.443444 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:01.444970 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:01.445453 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:01.445457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:01.445459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:01.445543 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:01.445843 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:01.445848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:01.445849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:01.446026 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:01.446137 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:01.446404 (+ 267us) server_negotiation.cc:299] Negotiation successful
0807 06:23:01.446516 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for oryx/127.15.254.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.254.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.2.keytab.
Entry for principal oryx/127.15.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.2.keytab.
I20260807 06:23:01.472353 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:0
--local_ip_for_outbound_sockets=127.15.254.2
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:33815
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.2.keytab
--principal=oryx/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:34831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:23:01.595427 27509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:23:01.595722 27509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:23:01.595768 27509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:23:01.599911 27509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:23:01.599990 27509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:23:01.600032 27509 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 06:23:01.600068 27509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:23:01.606412 27509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:34831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.2.keytab
--principal=oryx/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.254.62:33815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:23:01.607553 27509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:23:01.608610 27509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:23:01.619501 27524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:01.619457 27517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:01.619457 27518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:01.619901 27509 server_base.cc:1034] running on GCE node
I20260807 06:23:01.620457 27509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:23:01.621191 27509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:23:01.622382 27509 hybrid_clock.cc:648] HybridClock initialized: now 1786083781622359 us; error 34 us; skew 500 ppm
Aug 07 06:23:01 dist-test-slave-44d2 krb5kdc[27133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083781, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:23:01.625463 27509 init.cc:377] Logged in from keytab as oryx/127.15.254.2@KRBTEST.COM (short username oryx)
I20260807 06:23:01.626480 27509 webserver.cc:460] Webserver started at http://127.15.254.2:36377/ using document root <none> and password file <none>
I20260807 06:23:01.627120 27509 fs_manager.cc:359] Metadata directory not provided
I20260807 06:23:01.627223 27509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:23:01.627475 27509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:23:01.629768 27509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-1/data/instance:
uuid: "5d3c6d8875714934b60b33af8de46b82"
format_stamp: "Formatted at 2026-08-07 06:23:01 on dist-test-slave-44d2"
server_key: "815526ce536042988ba2c50647bfd5f4"
server_key_iv: "5e43857faa564930e7a39c8c3f1b0db3"
server_key_version: "encryptionkey@0"
I20260807 06:23:01.630407 27509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance:
uuid: "5d3c6d8875714934b60b33af8de46b82"
format_stamp: "Formatted at 2026-08-07 06:23:01 on dist-test-slave-44d2"
server_key: "815526ce536042988ba2c50647bfd5f4"
server_key_iv: "5e43857faa564930e7a39c8c3f1b0db3"
server_key_version: "encryptionkey@0"
I20260807 06:23:01.634685 27509 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:23:01.637542 27536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:01.638998 27509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:23:01.639173 27509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "5d3c6d8875714934b60b33af8de46b82"
format_stamp: "Formatted at 2026-08-07 06:23:01 on dist-test-slave-44d2"
server_key: "815526ce536042988ba2c50647bfd5f4"
server_key_iv: "5e43857faa564930e7a39c8c3f1b0db3"
server_key_version: "encryptionkey@0"
I20260807 06:23:01.639320 27509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:23:01.651531 27509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:23:01.655000 27509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:23:01.655231 27509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:23:01.656816 27509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:23:01.656893 27509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:01.656967 27509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:23:01.657019 27509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 06:23:01 dist-test-slave-44d2 krb5kdc[27133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083781, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM
I20260807 06:23:01.669082 27509 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:33975
I20260807 06:23:01.669175 27713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:33975 every 8 connection(s)
I20260807 06:23:01.670051 27509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:23:01.671269 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 27509
I20260807 06:23:01.671418 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance
I20260807 06:23:01.671758 16376 external_mini_cluster.cc:1428] Setting key ab7f0ce4794a68b2a188ef2c6d95ffde
I20260807 06:23:01.672720 27265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:01.657730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:44653 (local address 127.15.254.62:33815)
0807 06:23:01.657905 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:01.657912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:01.658852 (+ 940us) server_negotiation.cc:407] Connection header received
0807 06:23:01.660239 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:01.660243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:01.660337 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:01.660410 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:01.662558 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.663191 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.665187 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.665438 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.665572 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:01.668364 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:01.668368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:01.668371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:23:01.668408 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:01.670120 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:01.670690 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:01.670693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:01.670695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:01.670801 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:01.671241 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:01.671244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:01.671246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:01.671480 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:01.671564 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:01.672328 (+ 764us) server_negotiation.cc:299] Negotiation successful
0807 06:23:01.672522 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 06:23:01.673597 27553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:01.658130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:33815 (local address 127.15.254.2:44653)
0807 06:23:01.658642 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:01.658708 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:01.659935 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:01.660956 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:01.660968 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:01.661590 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:01.662362 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:01.662391 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.663347 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.663354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:01.665034 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:01.665048 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.665636 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.665640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:01.665824 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:01.666486 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:01.666503 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:01.668184 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:23:01.670239 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:23:01.670245 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:23:01.670260 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:23:01.670562 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:23:01.670910 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:23:01.670913 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:23:01.670915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:23:01.671124 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:23:01.671579 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:23:01.671587 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:23:01.671914 (+ 327us) client_negotiation.cc:769] Sending connection context
0807 06:23:01.672144 (+ 230us) client_negotiation.cc:240] Negotiation successful
0807 06:23:01.672540 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":112,"threads_started":1}
I20260807 06:23:01.675040 27714 heartbeater.cc:347] Connected to a master server at 127.15.254.62:33815
I20260807 06:23:01.675203 27714 heartbeater.cc:464] Registering TS with master...
I20260807 06:23:01.675832 27714 heartbeater.cc:511] Master 127.15.254.62:33815 requested a full tablet report, sending...
I20260807 06:23:01.677215 27195 ts_manager.cc:194] Registered new tserver with Master: 5d3c6d8875714934b60b33af8de46b82 (127.15.254.2:33975)
I20260807 06:23:01.678066 27195 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.254.2@KRBTEST.COM'} at 127.15.254.2:44653
WARNING: no policy specified for oryx/127.15.254.3@KRBTEST.COM; defaulting to no policy
I20260807 06:23:01.685897 27265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:01.677639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:49357 (local address 127.15.254.62:33815)
0807 06:23:01.677761 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:01.677765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:01.678039 (+ 274us) server_negotiation.cc:407] Connection header received
0807 06:23:01.678166 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:01.678171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:01.678232 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:01.678321 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:01.678913 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.679437 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.680428 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.680872 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.681011 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:01.682281 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:01.682285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:01.682287 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:23:01.682324 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:01.683994 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:01.684464 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:01.684468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:01.684471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:01.684601 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:01.684888 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:01.684894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:01.684897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:01.685259 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:01.685368 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:01.685669 (+ 301us) server_negotiation.cc:299] Negotiation successful
0807 06:23:01.685775 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 06:23:01.685896 27727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:01.677573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:33815 (local address 127.15.254.2:49357)
0807 06:23:01.677927 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:01.677942 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:01.678061 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:01.678371 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:01.678375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:01.678635 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:01.678780 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:01.678788 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.679588 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.679591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:01.680285 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:01.680296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.681022 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.681025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:01.681105 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:01.681727 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:01.681739 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:01.682117 (+ 378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:23:01.684142 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:23:01.684145 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:23:01.684147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:23:01.684357 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:23:01.684706 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:23:01.684709 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:23:01.684710 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:23:01.684788 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:23:01.685390 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:23:01.685393 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:23:01.685498 (+ 105us) client_negotiation.cc:769] Sending connection context
0807 06:23:01.685606 (+ 108us) client_negotiation.cc:240] Negotiation successful
0807 06:23:01.685735 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":135,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.3.keytab.
Entry for principal oryx/127.15.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.3.keytab.
I20260807 06:23:01.709151 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:0
--local_ip_for_outbound_sockets=127.15.254.3
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:33815
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.3.keytab
--principal=oryx/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:34831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:23:01.830818 27730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:23:01.831125 27730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:23:01.831172 27730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:23:01.835412 27730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:23:01.835515 27730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:23:01.835587 27730 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 06:23:01.835632 27730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:23:01.841495 27730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:34831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.3.keytab
--principal=oryx/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.254.62:33815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:23:01.842780 27730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:23:01.843928 27730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:23:01.854543 27743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:01.854543 27738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:01.854600 27739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:01.854758 27730 server_base.cc:1034] running on GCE node
I20260807 06:23:01.855335 27730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:23:01.856021 27730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:23:01.857234 27730 hybrid_clock.cc:648] HybridClock initialized: now 1786083781857216 us; error 30 us; skew 500 ppm
Aug 07 06:23:01 dist-test-slave-44d2 krb5kdc[27133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083781, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:23:01.860272 27730 init.cc:377] Logged in from keytab as oryx/127.15.254.3@KRBTEST.COM (short username oryx)
I20260807 06:23:01.861555 27730 webserver.cc:460] Webserver started at http://127.15.254.3:41439/ using document root <none> and password file <none>
I20260807 06:23:01.862286 27730 fs_manager.cc:359] Metadata directory not provided
I20260807 06:23:01.862391 27730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:23:01.862668 27730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:23:01.865132 27730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-2/data/instance:
uuid: "6a32044d9e4d4bd385dada5ce25fb0a9"
format_stamp: "Formatted at 2026-08-07 06:23:01 on dist-test-slave-44d2"
server_key: "74a7611f1490e1cb43467b18e0612b82"
server_key_iv: "bc6111914f4b05f78dd55ef15a99b3bb"
server_key_version: "encryptionkey@0"
I20260807 06:23:01.865789 27730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance:
uuid: "6a32044d9e4d4bd385dada5ce25fb0a9"
format_stamp: "Formatted at 2026-08-07 06:23:01 on dist-test-slave-44d2"
server_key: "74a7611f1490e1cb43467b18e0612b82"
server_key_iv: "bc6111914f4b05f78dd55ef15a99b3bb"
server_key_version: "encryptionkey@0"
I20260807 06:23:01.870134 27730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:23:01.873178 27754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:01.874810 27730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:23:01.875003 27730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "6a32044d9e4d4bd385dada5ce25fb0a9"
format_stamp: "Formatted at 2026-08-07 06:23:01 on dist-test-slave-44d2"
server_key: "74a7611f1490e1cb43467b18e0612b82"
server_key_iv: "bc6111914f4b05f78dd55ef15a99b3bb"
server_key_version: "encryptionkey@0"
I20260807 06:23:01.875151 27730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:23:01.898938 27730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:23:01.902547 27730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:23:01.902792 27730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:23:01.904419 27730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:23:01.904500 27730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:01.904631 27730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:23:01.904683 27730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 06:23:01 dist-test-slave-44d2 krb5kdc[27133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083781, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM
I20260807 06:23:01.916886 27730 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:40955
I20260807 06:23:01.916980 27917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:40955 every 8 connection(s)
I20260807 06:23:01.917992 27730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:23:01.919839 27265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:01.905576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:53191 (local address 127.15.254.62:33815)
0807 06:23:01.905762 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:01.905770 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:01.906653 (+ 883us) server_negotiation.cc:407] Connection header received
0807 06:23:01.907749 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:01.907755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:01.907821 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:01.907909 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:01.909684 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.910390 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.911787 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.912023 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.912154 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:01.915421 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:01.915426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:01.915431 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:23:01.915469 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:01.917301 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:01.917988 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:01.917992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:01.917994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:01.918117 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:01.918485 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:01.918490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:01.918491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:01.918779 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:01.918912 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:01.919561 (+ 649us) server_negotiation.cc:299] Negotiation successful
0807 06:23:01.919707 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 06:23:01.920874 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:01.905984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:33815 (local address 127.15.254.3:53191)
0807 06:23:01.906479 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:01.906534 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:01.907450 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:01.908433 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:01.908442 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:01.908928 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:01.909518 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:01.909534 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.910525 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.910529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:01.911653 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:01.911663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.912261 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.912267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:01.912447 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:01.913236 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:01.913253 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:01.915184 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:23:01.917434 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:23:01.917443 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:23:01.917461 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:23:01.917867 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:23:01.918210 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:23:01.918213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:23:01.918215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:23:01.918380 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:23:01.918977 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:23:01.918983 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:23:01.919311 (+ 328us) client_negotiation.cc:769] Sending connection context
0807 06:23:01.919486 (+ 175us) client_negotiation.cc:240] Negotiation successful
0807 06:23:01.919796 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":143,"threads_started":1}
I20260807 06:23:01.922413 27918 heartbeater.cc:347] Connected to a master server at 127.15.254.62:33815
I20260807 06:23:01.922652 27918 heartbeater.cc:464] Registering TS with master...
I20260807 06:23:01.923385 27918 heartbeater.cc:511] Master 127.15.254.62:33815 requested a full tablet report, sending...
I20260807 06:23:01.924724 27196 ts_manager.cc:194] Registered new tserver with Master: 6a32044d9e4d4bd385dada5ce25fb0a9 (127.15.254.3:40955)
I20260807 06:23:01.925760 27196 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.254.3@KRBTEST.COM'} at 127.15.254.3:53191
I20260807 06:23:01.928098 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 27730
I20260807 06:23:01.928239 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance
I20260807 06:23:01.928647 16376 external_mini_cluster.cc:1428] Setting key 5e8d4b353ebacbe1696c5132ca4b01a8
I20260807 06:23:01.931565 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:23:01.933787 27265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:01.924887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:35737 (local address 127.15.254.62:33815)
0807 06:23:01.925022 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:01.925027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:01.925304 (+ 277us) server_negotiation.cc:407] Connection header received
0807 06:23:01.925462 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:01.925465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:01.925510 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:01.925582 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:01.926173 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.926690 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.927925 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.928285 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.928375 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:01.930183 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:01.930187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:01.930190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:23:01.930217 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:01.931960 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:01.932414 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:01.932418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:01.932420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:01.932559 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:01.932878 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:01.932882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:01.932884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:01.933081 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:01.933191 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:01.933489 (+ 298us) server_negotiation.cc:299] Negotiation successful
0807 06:23:01.933620 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":48}
I20260807 06:23:01.933794 27929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:01.924753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:33815 (local address 127.15.254.3:35737)
0807 06:23:01.925199 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:01.925214 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:01.925331 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:01.925630 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:01.925634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:01.925909 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:01.926034 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:01.926040 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.926828 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.926831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:01.927762 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:01.927777 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.928595 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.928599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:01.928699 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:01.929443 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:01.929457 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:01.929995 (+ 538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:23:01.932087 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:23:01.932090 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:23:01.932093 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:23:01.932302 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:23:01.932678 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:23:01.932682 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:23:01.932684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:23:01.932779 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:23:01.933181 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:23:01.933185 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:23:01.933292 (+ 107us) client_negotiation.cc:769] Sending connection context
0807 06:23:01.933414 (+ 122us) client_negotiation.cc:240] Negotiation successful
0807 06:23:01.933576 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":112,"threads_started":1}
Aug 07 06:23:01 dist-test-slave-44d2 krb5kdc[27133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083781, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 06:23:01 dist-test-slave-44d2 krb5kdc[27133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:01 dist-test-slave-44d2 krb5kdc[27133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
W20260807 06:23:01.959443 27265 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:01.952666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59240 (local address 127.15.254.62:33815)
0807 06:23:01.952805 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:01.952808 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:01.952822 (+ 14us) server_negotiation.cc:407] Connection header received
0807 06:23:01.952866 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:01.952869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:01.952912 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:01.953035 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:01.953758 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.954324 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.955150 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.955360 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.955440 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:01.959195 (+ 3755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59240: BlockingRecv error: recv got EOF from 127.0.0.1:59240 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 07 06:23:01 dist-test-slave-44d2 krb5kdc[27133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083781, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM
I20260807 06:23:01.970417 27265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:01.961173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59246 (local address 127.15.254.62:33815)
0807 06:23:01.961308 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:01.961312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:01.961476 (+ 164us) server_negotiation.cc:407] Connection header received
0807 06:23:01.961636 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:01.961639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:01.961686 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:01.961762 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:01.962523 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.963104 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.964057 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.964280 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.964365 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:01.966971 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:01.966975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:01.966977 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:23:01.967011 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:01.968646 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:01.969151 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:01.969154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:01.969155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:01.969250 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:01.969497 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:01.969499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:01.969500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:01.969700 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:01.969809 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:01.970142 (+ 333us) server_negotiation.cc:299] Negotiation successful
0807 06:23:01.970260 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 06:23:01.973899 27196 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:23:01.977427 27196 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:23:01.993976 27955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:01.987583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56884 (local address 127.15.254.2:33975)
0807 06:23:01.988285 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:01.988305 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:01.988735 (+ 430us) server_negotiation.cc:407] Connection header received
0807 06:23:01.988979 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:01.988985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:01.989210 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:01.989376 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:23:01.989722 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.990948 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.992504 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.993186 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.993319 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:01.993518 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:01.993687 (+ 169us) server_negotiation.cc:299] Negotiation successful
0807 06:23:01.993797 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":125,"threads_started":1}
I20260807 06:23:01.993966 27953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:01.987412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:33975 (local address 127.0.0.1:56884)
0807 06:23:01.988561 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:01.988619 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:01.988763 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:01.989394 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:01.989400 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:01.989427 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:23:01.989582 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:01.989591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.991081 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.991085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:01.992326 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:01.992338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.993333 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.993336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:01.993450 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:01.993491 (+ 41us) client_negotiation.cc:769] Sending connection context
0807 06:23:01.993637 (+ 146us) client_negotiation.cc:240] Negotiation successful
0807 06:23:01.993803 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"mutex_wait_us":32,"spinlock_wait_cycles":6144,"thread_start_us":93,"threads_started":1}
I20260807 06:23:01.994011 27954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:01.987406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:46267 (local address 127.0.0.1:41454)
0807 06:23:01.988561 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:01.988619 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:01.988763 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:01.989499 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:01.989502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:01.989522 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:23:01.989633 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:01.989637 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.991121 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.991128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:01.992266 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:01.992277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.993475 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.993478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:01.993586 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:01.993613 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 06:23:01.993664 (+ 51us) client_negotiation.cc:240] Negotiation successful
0807 06:23:01.993804 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":36,"spinlock_wait_cycles":2560,"thread_start_us":108,"threads_started":1}
I20260807 06:23:01.994241 27956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:01.988060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:40955 (local address 127.0.0.1:45334)
0807 06:23:01.988927 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:01.988941 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:01.989032 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:01.989771 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:01.989774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:01.989792 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:23:01.989867 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:01.989872 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.990826 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.990832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:01.992534 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:01.992543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.993845 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.993848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:01.993939 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:01.993962 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 06:23:01.994035 (+ 73us) client_negotiation.cc:240] Negotiation successful
0807 06:23:01.994111 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":57,"thread_start_us":102,"threads_started":1}
I20260807 06:23:01.994798 27961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:01.988259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45334 (local address 127.15.254.3:40955)
0807 06:23:01.989297 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:01.989332 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:01.989355 (+ 23us) server_negotiation.cc:407] Connection header received
0807 06:23:01.989430 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:01.989435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:01.989619 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:01.989800 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:23:01.989965 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.990704 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.992694 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.993505 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.993605 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:01.993783 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:01.994512 (+ 729us) server_negotiation.cc:299] Negotiation successful
0807 06:23:01.994619 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":946,"thread_start_us":112,"threads_started":1}
I20260807 06:23:01.995463 27958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:01.988259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41454 (local address 127.15.254.1:46267)
0807 06:23:01.989080 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:01.989087 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:01.989109 (+ 22us) server_negotiation.cc:407] Connection header received
0807 06:23:01.989181 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:01.989185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:01.989355 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:01.989823 (+ 468us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:23:01.989965 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.991028 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.992399 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:01.993318 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:01.994452 (+ 1134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:01.995177 (+ 725us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:01.995228 (+ 51us) server_negotiation.cc:299] Negotiation successful
0807 06:23:01.995329 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":103,"threads_started":1}
I20260807 06:23:01.997740 27620 tablet_service.cc:1467] Processing CreateTablet for tablet fb37f11d58ad4da9996e973564c849c1 (DEFAULT_TABLE table=test-table [id=d702c27ef7944e99870d5ed977b2632a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:23:01.997766 27827 tablet_service.cc:1467] Processing CreateTablet for tablet fb37f11d58ad4da9996e973564c849c1 (DEFAULT_TABLE table=test-table [id=d702c27ef7944e99870d5ed977b2632a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:23:01.998483 27394 tablet_service.cc:1467] Processing CreateTablet for tablet fb37f11d58ad4da9996e973564c849c1 (DEFAULT_TABLE table=test-table [id=d702c27ef7944e99870d5ed977b2632a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:23:01.998973 27827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb37f11d58ad4da9996e973564c849c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:23:01.998970 27620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb37f11d58ad4da9996e973564c849c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:23:01.999617 27394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb37f11d58ad4da9996e973564c849c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:23:02.007179 27962 tablet_bootstrap.cc:492] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82: Bootstrap starting.
I20260807 06:23:02.008467 27964 tablet_bootstrap.cc:492] T fb37f11d58ad4da9996e973564c849c1 P 2d41a707b8bc4282ae3a5f954f607358: Bootstrap starting.
I20260807 06:23:02.008745 27963 tablet_bootstrap.cc:492] T fb37f11d58ad4da9996e973564c849c1 P 6a32044d9e4d4bd385dada5ce25fb0a9: Bootstrap starting.
I20260807 06:23:02.009845 27962 tablet_bootstrap.cc:654] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82: Neither blocks nor log segments found. Creating new log.
I20260807 06:23:02.010849 27963 tablet_bootstrap.cc:654] T fb37f11d58ad4da9996e973564c849c1 P 6a32044d9e4d4bd385dada5ce25fb0a9: Neither blocks nor log segments found. Creating new log.
I20260807 06:23:02.010849 27964 tablet_bootstrap.cc:654] T fb37f11d58ad4da9996e973564c849c1 P 2d41a707b8bc4282ae3a5f954f607358: Neither blocks nor log segments found. Creating new log.
I20260807 06:23:02.010936 27962 log.cc:824] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82: Log is configured to *not* fsync() on all Append() calls
I20260807 06:23:02.012104 27964 log.cc:824] T fb37f11d58ad4da9996e973564c849c1 P 2d41a707b8bc4282ae3a5f954f607358: Log is configured to *not* fsync() on all Append() calls
I20260807 06:23:02.012104 27963 log.cc:824] T fb37f11d58ad4da9996e973564c849c1 P 6a32044d9e4d4bd385dada5ce25fb0a9: Log is configured to *not* fsync() on all Append() calls
I20260807 06:23:02.013394 27962 tablet_bootstrap.cc:492] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82: No bootstrap required, opened a new log
I20260807 06:23:02.013655 27962 ts_tablet_manager.cc:1397] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260807 06:23:02.014057 27963 tablet_bootstrap.cc:492] T fb37f11d58ad4da9996e973564c849c1 P 6a32044d9e4d4bd385dada5ce25fb0a9: No bootstrap required, opened a new log
I20260807 06:23:02.014222 27963 ts_tablet_manager.cc:1397] T fb37f11d58ad4da9996e973564c849c1 P 6a32044d9e4d4bd385dada5ce25fb0a9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20260807 06:23:02.017704 27962 raft_consensus.cc:348] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.017692 27963 raft_consensus.cc:348] T fb37f11d58ad4da9996e973564c849c1 P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.017954 27963 raft_consensus.cc:374] T fb37f11d58ad4da9996e973564c849c1 P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:23:02.017961 27962 raft_consensus.cc:374] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:23:02.018026 27962 raft_consensus.cc:729] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d3c6d8875714934b60b33af8de46b82, State: Initialized, Role: FOLLOWER
I20260807 06:23:02.018043 27963 raft_consensus.cc:729] T fb37f11d58ad4da9996e973564c849c1 P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a32044d9e4d4bd385dada5ce25fb0a9, State: Initialized, Role: FOLLOWER
I20260807 06:23:02.018517 27962 consensus_queue.cc:260] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.018522 27963 consensus_queue.cc:260] T fb37f11d58ad4da9996e973564c849c1 P 6a32044d9e4d4bd385dada5ce25fb0a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.019474 27918 heartbeater.cc:503] Master 127.15.254.62:33815 was elected leader, sending a full tablet report...
I20260807 06:23:02.019517 27714 heartbeater.cc:503] Master 127.15.254.62:33815 was elected leader, sending a full tablet report...
I20260807 06:23:02.019634 27962 ts_tablet_manager.cc:1428] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260807 06:23:02.019634 27963 ts_tablet_manager.cc:1428] T fb37f11d58ad4da9996e973564c849c1 P 6a32044d9e4d4bd385dada5ce25fb0a9: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20260807 06:23:02.020306 27964 tablet_bootstrap.cc:492] T fb37f11d58ad4da9996e973564c849c1 P 2d41a707b8bc4282ae3a5f954f607358: No bootstrap required, opened a new log
I20260807 06:23:02.020587 27964 ts_tablet_manager.cc:1397] T fb37f11d58ad4da9996e973564c849c1 P 2d41a707b8bc4282ae3a5f954f607358: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.002s
I20260807 06:23:02.024679 27964 raft_consensus.cc:348] T fb37f11d58ad4da9996e973564c849c1 P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.025027 27964 raft_consensus.cc:374] T fb37f11d58ad4da9996e973564c849c1 P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:23:02.025106 27964 raft_consensus.cc:729] T fb37f11d58ad4da9996e973564c849c1 P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d41a707b8bc4282ae3a5f954f607358, State: Initialized, Role: FOLLOWER
I20260807 06:23:02.025588 27964 consensus_queue.cc:260] T fb37f11d58ad4da9996e973564c849c1 P 2d41a707b8bc4282ae3a5f954f607358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.026257 27499 heartbeater.cc:503] Master 127.15.254.62:33815 was elected leader, sending a full tablet report...
I20260807 06:23:02.026587 27964 ts_tablet_manager.cc:1428] T fb37f11d58ad4da9996e973564c849c1 P 2d41a707b8bc4282ae3a5f954f607358: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260807 06:23:02.095975 27970 raft_consensus.cc:482] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:23:02.096221 27970 raft_consensus.cc:504] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.097416 27970 leader_election.cc:288] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d41a707b8bc4282ae3a5f954f607358 (127.15.254.1:46267), 6a32044d9e4d4bd385dada5ce25fb0a9 (127.15.254.3:40955)
I20260807 06:23:02.102087 27553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:02.097679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:40955 (local address 127.15.254.2:39065)
0807 06:23:02.097880 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:02.097898 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:02.097993 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:02.098423 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:02.098426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:02.098446 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:23:02.098588 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:02.098598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.099699 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.099703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:02.100659 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:02.100670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.101651 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.101655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:02.101772 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:02.101795 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 06:23:02.101872 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 06:23:02.101945 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":1}
I20260807 06:23:02.102156 27961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:02.097824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:39065 (local address 127.15.254.3:40955)
0807 06:23:02.097955 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:02.097960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:02.097976 (+ 16us) server_negotiation.cc:407] Connection header received
0807 06:23:02.098154 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:02.098157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:02.098200 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:02.098294 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:23:02.098726 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.099586 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.100793 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.101417 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.101534 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:02.101586 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:02.101914 (+ 328us) server_negotiation.cc:299] Negotiation successful
0807 06:23:02.102018 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 06:23:02.102685 27958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:02.097870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:58307 (local address 127.15.254.1:46267)
0807 06:23:02.097994 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:02.097998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:02.098544 (+ 546us) server_negotiation.cc:407] Connection header received
0807 06:23:02.098653 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:02.098656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:02.098709 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:02.098789 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:23:02.099505 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.100267 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.101470 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.101942 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.102056 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:02.102104 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:02.102444 (+ 340us) server_negotiation.cc:299] Negotiation successful
0807 06:23:02.102527 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 06:23:02.102941 27855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb37f11d58ad4da9996e973564c849c1" candidate_uuid: "5d3c6d8875714934b60b33af8de46b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" is_pre_election: true
I20260807 06:23:02.103055 27974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:02.097679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:46267 (local address 127.15.254.2:58307)
0807 06:23:02.098430 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:02.098444 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:02.098560 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:02.098825 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:02.098828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:02.098846 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:23:02.099375 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:02.099384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.100393 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.100396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:02.101333 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:02.101343 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.102183 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.102187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:02.102293 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:02.102322 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 06:23:02.102830 (+ 508us) client_negotiation.cc:240] Negotiation successful
0807 06:23:02.102902 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":370,"thread_start_us":166,"threads_started":1}
I20260807 06:23:02.103288 27855 raft_consensus.cc:2454] T fb37f11d58ad4da9996e973564c849c1 P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d3c6d8875714934b60b33af8de46b82 in term 0.
I20260807 06:23:02.103739 27429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb37f11d58ad4da9996e973564c849c1" candidate_uuid: "5d3c6d8875714934b60b33af8de46b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d41a707b8bc4282ae3a5f954f607358" is_pre_election: true
I20260807 06:23:02.103850 27542 leader_election.cc:302] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d3c6d8875714934b60b33af8de46b82, 6a32044d9e4d4bd385dada5ce25fb0a9; no voters:
I20260807 06:23:02.104056 27429 raft_consensus.cc:2454] T fb37f11d58ad4da9996e973564c849c1 P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d3c6d8875714934b60b33af8de46b82 in term 0.
I20260807 06:23:02.104166 27970 raft_consensus.cc:2781] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:23:02.104266 27970 raft_consensus.cc:482] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:23:02.104316 27970 raft_consensus.cc:3037] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:23:02.105665 27970 raft_consensus.cc:504] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.106045 27970 leader_election.cc:288] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82 [CANDIDATE]: Term 1 election: Requested vote from peers 2d41a707b8bc4282ae3a5f954f607358 (127.15.254.1:46267), 6a32044d9e4d4bd385dada5ce25fb0a9 (127.15.254.3:40955)
I20260807 06:23:02.106400 27429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb37f11d58ad4da9996e973564c849c1" candidate_uuid: "5d3c6d8875714934b60b33af8de46b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d41a707b8bc4282ae3a5f954f607358"
I20260807 06:23:02.106467 27855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb37f11d58ad4da9996e973564c849c1" candidate_uuid: "5d3c6d8875714934b60b33af8de46b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9"
I20260807 06:23:02.106530 27429 raft_consensus.cc:3037] T fb37f11d58ad4da9996e973564c849c1 P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:23:02.106627 27855 raft_consensus.cc:3037] T fb37f11d58ad4da9996e973564c849c1 P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:23:02.107702 27855 raft_consensus.cc:2454] T fb37f11d58ad4da9996e973564c849c1 P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d3c6d8875714934b60b33af8de46b82 in term 1.
I20260807 06:23:02.107717 27429 raft_consensus.cc:2454] T fb37f11d58ad4da9996e973564c849c1 P 2d41a707b8bc4282ae3a5f954f607358 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d3c6d8875714934b60b33af8de46b82 in term 1.
I20260807 06:23:02.108042 27542 leader_election.cc:302] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d3c6d8875714934b60b33af8de46b82, 6a32044d9e4d4bd385dada5ce25fb0a9; no voters:
I20260807 06:23:02.108253 27970 raft_consensus.cc:2781] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:23:02.108532 27970 raft_consensus.cc:686] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82 [term 1 LEADER]: Becoming Leader. State: Replica: 5d3c6d8875714934b60b33af8de46b82, State: Running, Role: LEADER
I20260807 06:23:02.108884 27970 consensus_queue.cc:237] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.112250 27194 catalog_manager.cc:5495] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d3c6d8875714934b60b33af8de46b82 (127.15.254.2). New cstate: current_term: 1 leader_uuid: "5d3c6d8875714934b60b33af8de46b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } health_report { overall_health: UNKNOWN } } }
I20260807 06:23:02.125970 27955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:02.122167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56898 (local address 127.15.254.2:33975)
0807 06:23:02.122321 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:02.122326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:02.122342 (+ 16us) server_negotiation.cc:407] Connection header received
0807 06:23:02.122515 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:02.122517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:02.122576 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:02.122695 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:23:02.122961 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.123491 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.124372 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.124652 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.124762 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:02.125070 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:23:02.125538 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:23:02.125668 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:02.125747 (+ 79us) server_negotiation.cc:299] Negotiation successful
0807 06:23:02.125811 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 06:23:02.132371 27429 raft_consensus.cc:1261] T fb37f11d58ad4da9996e973564c849c1 P 2d41a707b8bc4282ae3a5f954f607358 [term 1 FOLLOWER]: Refusing update from remote peer 5d3c6d8875714934b60b33af8de46b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:23:02.132741 27855 raft_consensus.cc:1261] T fb37f11d58ad4da9996e973564c849c1 P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 1 FOLLOWER]: Refusing update from remote peer 5d3c6d8875714934b60b33af8de46b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:23:02.133226 27977 consensus_queue.cc:1035] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:23:02.133415 27970 consensus_queue.cc:1035] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:23:02.141208 27980 mvcc.cc:205] Tried to move back new op lower bound from 7315799171606126592 to 7315799171520634880. Current Snapshot: MvccSnapshot[applied={T|T < 7315799171606126592}]
I20260807 06:23:02.144853 27981 mvcc.cc:205] Tried to move back new op lower bound from 7315799171606126592 to 7315799171520634880. Current Snapshot: MvccSnapshot[applied={T|T < 7315799171606126592}]
I20260807 06:23:02.151233 27194 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:23:02.151438 27194 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:23:02.154088 27194 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7: Sending DeleteTablet for 3 replicas of tablet fb37f11d58ad4da9996e973564c849c1
I20260807 06:23:02.154894 27394 tablet_service.cc:1514] Processing DeleteTablet for tablet fb37f11d58ad4da9996e973564c849c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:23:02 UTC) from {username='oryx', principal='oryx/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:41454
I20260807 06:23:02.154942 27620 tablet_service.cc:1514] Processing DeleteTablet for tablet fb37f11d58ad4da9996e973564c849c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:23:02 UTC) from {username='oryx', principal='oryx/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:56884
I20260807 06:23:02.155018 27827 tablet_service.cc:1514] Processing DeleteTablet for tablet fb37f11d58ad4da9996e973564c849c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:23:02 UTC) from {username='oryx', principal='oryx/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:45334
I20260807 06:23:02.155581 28000 tablet_replica.cc:330] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82: stopping tablet replica
I20260807 06:23:02.155581 27997 tablet_replica.cc:330] T fb37f11d58ad4da9996e973564c849c1 P 2d41a707b8bc4282ae3a5f954f607358: stopping tablet replica
I20260807 06:23:02.155902 27997 raft_consensus.cc:2229] T fb37f11d58ad4da9996e973564c849c1 P 2d41a707b8bc4282ae3a5f954f607358 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:23:02.156112 28000 raft_consensus.cc:2229] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:23:02.156255 27997 raft_consensus.cc:2258] T fb37f11d58ad4da9996e973564c849c1 P 2d41a707b8bc4282ae3a5f954f607358 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:23:02.156378 27194 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 06:23:02.156661 27998 tablet_replica.cc:330] T fb37f11d58ad4da9996e973564c849c1 P 6a32044d9e4d4bd385dada5ce25fb0a9: stopping tablet replica
W20260807 06:23:02.156873 27194 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:23:02.156939 27998 raft_consensus.cc:2229] T fb37f11d58ad4da9996e973564c849c1 P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:23:02.157253 27998 raft_consensus.cc:2258] T fb37f11d58ad4da9996e973564c849c1 P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:23:02.158805 27997 ts_tablet_manager.cc:1905] T fb37f11d58ad4da9996e973564c849c1 P 2d41a707b8bc4282ae3a5f954f607358: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:23:02.158805 27998 ts_tablet_manager.cc:1905] T fb37f11d58ad4da9996e973564c849c1 P 6a32044d9e4d4bd385dada5ce25fb0a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:23:02.162052 27997 ts_tablet_manager.cc:1918] T fb37f11d58ad4da9996e973564c849c1 P 2d41a707b8bc4282ae3a5f954f607358: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:23:02.162262 27997 log.cc:1196] T fb37f11d58ad4da9996e973564c849c1 P 2d41a707b8bc4282ae3a5f954f607358: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-0/wal/wals/fb37f11d58ad4da9996e973564c849c1
I20260807 06:23:02.162308 27998 ts_tablet_manager.cc:1918] T fb37f11d58ad4da9996e973564c849c1 P 6a32044d9e4d4bd385dada5ce25fb0a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:23:02.162464 27998 log.cc:1196] T fb37f11d58ad4da9996e973564c849c1 P 6a32044d9e4d4bd385dada5ce25fb0a9: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/fb37f11d58ad4da9996e973564c849c1
I20260807 06:23:02.162820 27997 ts_tablet_manager.cc:1939] T fb37f11d58ad4da9996e973564c849c1 P 2d41a707b8bc4282ae3a5f954f607358: Deleting consensus metadata
I20260807 06:23:02.162889 27998 ts_tablet_manager.cc:1939] T fb37f11d58ad4da9996e973564c849c1 P 6a32044d9e4d4bd385dada5ce25fb0a9: Deleting consensus metadata
I20260807 06:23:02.164139 27394 tablet_service.cc:1467] Processing CreateTablet for tablet dba959bf3c72480b8591702901953d5e (DEFAULT_TABLE table=test-table [id=8f8fe5e5dd99456abacdd6cf932160d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:23:02.164427 27394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dba959bf3c72480b8591702901953d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:23:02.164579 27620 tablet_service.cc:1467] Processing CreateTablet for tablet dba959bf3c72480b8591702901953d5e (DEFAULT_TABLE table=test-table [id=8f8fe5e5dd99456abacdd6cf932160d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:23:02.164873 27620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dba959bf3c72480b8591702901953d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:23:02.166731 27964 tablet_bootstrap.cc:492] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358: Bootstrap starting.
I20260807 06:23:02.167600 27176 catalog_manager.cc:4841] TS 2d41a707b8bc4282ae3a5f954f607358 (127.15.254.1:46267): tablet fb37f11d58ad4da9996e973564c849c1 (table test-table [id=d702c27ef7944e99870d5ed977b2632a]) successfully deleted
I20260807 06:23:02.167682 27178 catalog_manager.cc:4841] TS 6a32044d9e4d4bd385dada5ce25fb0a9 (127.15.254.3:40955): tablet fb37f11d58ad4da9996e973564c849c1 (table test-table [id=d702c27ef7944e99870d5ed977b2632a]) successfully deleted
I20260807 06:23:02.167865 27964 tablet_bootstrap.cc:654] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358: Neither blocks nor log segments found. Creating new log.
I20260807 06:23:02.168206 27827 tablet_service.cc:1467] Processing CreateTablet for tablet dba959bf3c72480b8591702901953d5e (DEFAULT_TABLE table=test-table [id=8f8fe5e5dd99456abacdd6cf932160d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:23:02.168463 27827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dba959bf3c72480b8591702901953d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:23:02.168576 27962 tablet_bootstrap.cc:492] T dba959bf3c72480b8591702901953d5e P 5d3c6d8875714934b60b33af8de46b82: Bootstrap starting.
I20260807 06:23:02.169072 28000 raft_consensus.cc:2258] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:23:02.169674 27962 tablet_bootstrap.cc:654] T dba959bf3c72480b8591702901953d5e P 5d3c6d8875714934b60b33af8de46b82: Neither blocks nor log segments found. Creating new log.
I20260807 06:23:02.170149 27964 tablet_bootstrap.cc:492] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358: No bootstrap required, opened a new log
I20260807 06:23:02.170224 27964 ts_tablet_manager.cc:1397] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260807 06:23:02.170744 27964 raft_consensus.cc:348] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.170862 27963 tablet_bootstrap.cc:492] T dba959bf3c72480b8591702901953d5e P 6a32044d9e4d4bd385dada5ce25fb0a9: Bootstrap starting.
I20260807 06:23:02.170868 27964 raft_consensus.cc:374] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:23:02.170907 27964 raft_consensus.cc:729] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d41a707b8bc4282ae3a5f954f607358, State: Initialized, Role: FOLLOWER
I20260807 06:23:02.171028 27964 consensus_queue.cc:260] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.171820 27962 tablet_bootstrap.cc:492] T dba959bf3c72480b8591702901953d5e P 5d3c6d8875714934b60b33af8de46b82: No bootstrap required, opened a new log
I20260807 06:23:02.171911 27963 tablet_bootstrap.cc:654] T dba959bf3c72480b8591702901953d5e P 6a32044d9e4d4bd385dada5ce25fb0a9: Neither blocks nor log segments found. Creating new log.
I20260807 06:23:02.171929 27962 ts_tablet_manager.cc:1397] T dba959bf3c72480b8591702901953d5e P 5d3c6d8875714934b60b33af8de46b82: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 06:23:02.172459 27964 ts_tablet_manager.cc:1428] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260807 06:23:02.172434 27962 raft_consensus.cc:348] T dba959bf3c72480b8591702901953d5e P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.172614 27962 raft_consensus.cc:374] T dba959bf3c72480b8591702901953d5e P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:23:02.172654 27962 raft_consensus.cc:729] T dba959bf3c72480b8591702901953d5e P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d3c6d8875714934b60b33af8de46b82, State: Initialized, Role: FOLLOWER
I20260807 06:23:02.172816 27962 consensus_queue.cc:260] T dba959bf3c72480b8591702901953d5e P 5d3c6d8875714934b60b33af8de46b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.173194 27962 ts_tablet_manager.cc:1428] T dba959bf3c72480b8591702901953d5e P 5d3c6d8875714934b60b33af8de46b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 06:23:02.175166 27963 tablet_bootstrap.cc:492] T dba959bf3c72480b8591702901953d5e P 6a32044d9e4d4bd385dada5ce25fb0a9: No bootstrap required, opened a new log
I20260807 06:23:02.175287 27963 ts_tablet_manager.cc:1397] T dba959bf3c72480b8591702901953d5e P 6a32044d9e4d4bd385dada5ce25fb0a9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260807 06:23:02.175791 27963 raft_consensus.cc:348] T dba959bf3c72480b8591702901953d5e P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.175933 27963 raft_consensus.cc:374] T dba959bf3c72480b8591702901953d5e P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:23:02.175980 27963 raft_consensus.cc:729] T dba959bf3c72480b8591702901953d5e P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a32044d9e4d4bd385dada5ce25fb0a9, State: Initialized, Role: FOLLOWER
I20260807 06:23:02.176133 27963 consensus_queue.cc:260] T dba959bf3c72480b8591702901953d5e P 6a32044d9e4d4bd385dada5ce25fb0a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.176368 28000 ts_tablet_manager.cc:1905] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:23:02.176432 27963 ts_tablet_manager.cc:1428] T dba959bf3c72480b8591702901953d5e P 6a32044d9e4d4bd385dada5ce25fb0a9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260807 06:23:02.179441 28000 ts_tablet_manager.cc:1918] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:23:02.179569 28000 log.cc:1196] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/fb37f11d58ad4da9996e973564c849c1
I20260807 06:23:02.179883 28000 ts_tablet_manager.cc:1939] T fb37f11d58ad4da9996e973564c849c1 P 5d3c6d8875714934b60b33af8de46b82: Deleting consensus metadata
I20260807 06:23:02.181118 27175 catalog_manager.cc:4841] TS 5d3c6d8875714934b60b33af8de46b82 (127.15.254.2:33975): tablet fb37f11d58ad4da9996e973564c849c1 (table test-table [id=d702c27ef7944e99870d5ed977b2632a]) successfully deleted
W20260807 06:23:02.182222 27500 tablet.cc:2369] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:23:02.314975 27973 raft_consensus.cc:482] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:23:02.315143 27973 raft_consensus.cc:504] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.316455 27973 leader_election.cc:288] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d3c6d8875714934b60b33af8de46b82 (127.15.254.2:33975), 6a32044d9e4d4bd385dada5ce25fb0a9 (127.15.254.3:40955)
I20260807 06:23:02.321614 27955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:02.317088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:37313 (local address 127.15.254.2:33975)
0807 06:23:02.317225 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:02.317230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:02.317246 (+ 16us) server_negotiation.cc:407] Connection header received
0807 06:23:02.317434 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:02.317438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:02.317500 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:02.317560 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:23:02.318051 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.318608 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.320315 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.320849 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.321015 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:02.321088 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:02.321327 (+ 239us) server_negotiation.cc:299] Negotiation successful
0807 06:23:02.321442 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 06:23:02.321640 28008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:02.316805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:40955 (local address 127.15.254.1:36339)
0807 06:23:02.317464 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:02.317477 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:02.317582 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:02.317832 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:02.317834 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:02.317854 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:23:02.317987 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:02.317997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.319026 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.319029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:02.320156 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:02.320163 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.321167 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.321170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:02.321263 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:02.321299 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 06:23:02.321371 (+ 72us) client_negotiation.cc:240] Negotiation successful
0807 06:23:02.321448 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":4,"thread_start_us":110,"threads_started":1}
I20260807 06:23:02.321653 27961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:02.317174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:36339 (local address 127.15.254.3:40955)
0807 06:23:02.317333 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:02.317337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:02.317579 (+ 242us) server_negotiation.cc:407] Connection header received
0807 06:23:02.317666 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:02.317670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:02.317720 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:02.317806 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:23:02.318114 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.318900 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.320306 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.320993 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.321128 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:02.321182 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:02.321409 (+ 227us) server_negotiation.cc:299] Negotiation successful
0807 06:23:02.321497 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 06:23:02.322425 28007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:02.316794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:33975 (local address 127.15.254.1:37313)
0807 06:23:02.317131 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:02.317150 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:02.317274 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:02.317720 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:02.317722 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:02.317739 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:23:02.317901 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:02.317908 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.318739 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.318741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:02.320124 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:02.320140 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.321034 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.321037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:02.321165 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:02.321192 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 06:23:02.322090 (+ 898us) client_negotiation.cc:240] Negotiation successful
0807 06:23:02.322184 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":79,"threads_started":1}
I20260807 06:23:02.322460 27855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba959bf3c72480b8591702901953d5e" candidate_uuid: "2d41a707b8bc4282ae3a5f954f607358" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" is_pre_election: true
I20260807 06:23:02.322655 27855 raft_consensus.cc:2454] T dba959bf3c72480b8591702901953d5e P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d41a707b8bc4282ae3a5f954f607358 in term 0.
I20260807 06:23:02.323077 27647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba959bf3c72480b8591702901953d5e" candidate_uuid: "2d41a707b8bc4282ae3a5f954f607358" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3c6d8875714934b60b33af8de46b82" is_pre_election: true
I20260807 06:23:02.323107 27331 leader_election.cc:302] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d41a707b8bc4282ae3a5f954f607358, 6a32044d9e4d4bd385dada5ce25fb0a9; no voters:
I20260807 06:23:02.323352 27647 raft_consensus.cc:2454] T dba959bf3c72480b8591702901953d5e P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d41a707b8bc4282ae3a5f954f607358 in term 0.
I20260807 06:23:02.323480 27973 raft_consensus.cc:2781] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:23:02.323544 27973 raft_consensus.cc:482] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:23:02.323593 27973 raft_consensus.cc:3037] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:23:02.324864 27973 raft_consensus.cc:504] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.325410 27973 leader_election.cc:288] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358 [CANDIDATE]: Term 1 election: Requested vote from peers 5d3c6d8875714934b60b33af8de46b82 (127.15.254.2:33975), 6a32044d9e4d4bd385dada5ce25fb0a9 (127.15.254.3:40955)
I20260807 06:23:02.325796 27647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba959bf3c72480b8591702901953d5e" candidate_uuid: "2d41a707b8bc4282ae3a5f954f607358" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3c6d8875714934b60b33af8de46b82"
I20260807 06:23:02.325845 27855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba959bf3c72480b8591702901953d5e" candidate_uuid: "2d41a707b8bc4282ae3a5f954f607358" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9"
I20260807 06:23:02.325922 27647 raft_consensus.cc:3037] T dba959bf3c72480b8591702901953d5e P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:23:02.325981 27855 raft_consensus.cc:3037] T dba959bf3c72480b8591702901953d5e P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:23:02.327210 27855 raft_consensus.cc:2454] T dba959bf3c72480b8591702901953d5e P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d41a707b8bc4282ae3a5f954f607358 in term 1.
I20260807 06:23:02.327514 27647 raft_consensus.cc:2454] T dba959bf3c72480b8591702901953d5e P 5d3c6d8875714934b60b33af8de46b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d41a707b8bc4282ae3a5f954f607358 in term 1.
I20260807 06:23:02.327536 27331 leader_election.cc:302] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d41a707b8bc4282ae3a5f954f607358, 6a32044d9e4d4bd385dada5ce25fb0a9; no voters:
I20260807 06:23:02.327777 27973 raft_consensus.cc:2781] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:23:02.328034 27973 raft_consensus.cc:686] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358 [term 1 LEADER]: Becoming Leader. State: Replica: 2d41a707b8bc4282ae3a5f954f607358, State: Running, Role: LEADER
I20260807 06:23:02.328349 27973 consensus_queue.cc:237] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.330866 27194 catalog_manager.cc:5495] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d41a707b8bc4282ae3a5f954f607358 (127.15.254.1). New cstate: current_term: 1 leader_uuid: "2d41a707b8bc4282ae3a5f954f607358" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } health_report { overall_health: UNKNOWN } } }
Aug 07 06:23:02 dist-test-slave-44d2 krb5kdc[27133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083781, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.62@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM
I20260807 06:23:02.391693 28020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:02.382420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:33815 (local address 127.0.0.1:59250)
0807 06:23:02.382765 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:02.382778 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:02.382973 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:02.383251 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:02.383254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:02.383686 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:02.383778 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:02.383785 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.384595 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.384600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:02.385157 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:02.385165 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.385624 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.385627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:02.385746 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:02.386442 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:02.386459 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:02.388128 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:23:02.389897 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:23:02.389908 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:23:02.389915 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:23:02.390158 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:23:02.390447 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:23:02.390451 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:23:02.390453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:23:02.390537 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:23:02.391004 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:23:02.391043 (+ 39us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:23:02.391149 (+ 106us) client_negotiation.cc:769] Sending connection context
0807 06:23:02.391272 (+ 123us) client_negotiation.cc:240] Negotiation successful
0807 06:23:02.391460 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":20,"thread_start_us":136,"threads_started":1}
I20260807 06:23:02.391693 27265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:02.382463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59250 (local address 127.15.254.62:33815)
0807 06:23:02.382620 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:02.382626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:02.382857 (+ 231us) server_negotiation.cc:407] Connection header received
0807 06:23:02.383084 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:02.383088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:02.383138 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:02.383226 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:02.383897 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.384451 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.385278 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.385507 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.385617 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:02.388282 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:02.388285 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:02.388287 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:23:02.388308 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:02.389753 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:02.390265 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:02.390270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:02.390272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:02.390351 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:02.390635 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:02.390638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:02.390640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:02.390819 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:02.390923 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:02.391330 (+ 407us) server_negotiation.cc:299] Negotiation successful
0807 06:23:02.391518 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":55}
I20260807 06:23:02.395965 27194 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:59250:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 06:23:02.396733 27194 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:23:02.402606 27394 tablet_service.cc:1467] Processing CreateTablet for tablet a9dc9360dfa64f1a842b592b86ca8a37 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b953a4d8c8674f22b3619f40b4ac8772]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 06:23:02.402679 27620 tablet_service.cc:1467] Processing CreateTablet for tablet a9dc9360dfa64f1a842b592b86ca8a37 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b953a4d8c8674f22b3619f40b4ac8772]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 06:23:02.403077 27394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9dc9360dfa64f1a842b592b86ca8a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:23:02.403086 27620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9dc9360dfa64f1a842b592b86ca8a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:23:02.405560 27962 tablet_bootstrap.cc:492] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82: Bootstrap starting.
I20260807 06:23:02.405627 27964 tablet_bootstrap.cc:492] T a9dc9360dfa64f1a842b592b86ca8a37 P 2d41a707b8bc4282ae3a5f954f607358: Bootstrap starting.
I20260807 06:23:02.406075 27827 tablet_service.cc:1467] Processing CreateTablet for tablet a9dc9360dfa64f1a842b592b86ca8a37 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b953a4d8c8674f22b3619f40b4ac8772]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 06:23:02.406508 27827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9dc9360dfa64f1a842b592b86ca8a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:23:02.406514 27962 tablet_bootstrap.cc:654] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82: Neither blocks nor log segments found. Creating new log.
I20260807 06:23:02.406738 27964 tablet_bootstrap.cc:654] T a9dc9360dfa64f1a842b592b86ca8a37 P 2d41a707b8bc4282ae3a5f954f607358: Neither blocks nor log segments found. Creating new log.
I20260807 06:23:02.407797 27962 tablet_bootstrap.cc:492] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82: No bootstrap required, opened a new log
I20260807 06:23:02.407927 27962 ts_tablet_manager.cc:1397] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 06:23:02.408079 27964 tablet_bootstrap.cc:492] T a9dc9360dfa64f1a842b592b86ca8a37 P 2d41a707b8bc4282ae3a5f954f607358: No bootstrap required, opened a new log
I20260807 06:23:02.408182 27964 ts_tablet_manager.cc:1397] T a9dc9360dfa64f1a842b592b86ca8a37 P 2d41a707b8bc4282ae3a5f954f607358: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 06:23:02.408378 27962 raft_consensus.cc:348] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.408545 27962 raft_consensus.cc:374] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:23:02.408582 27962 raft_consensus.cc:729] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d3c6d8875714934b60b33af8de46b82, State: Initialized, Role: FOLLOWER
I20260807 06:23:02.408674 27964 raft_consensus.cc:348] T a9dc9360dfa64f1a842b592b86ca8a37 P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.408787 27964 raft_consensus.cc:374] T a9dc9360dfa64f1a842b592b86ca8a37 P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:23:02.408723 27962 consensus_queue.cc:260] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.408821 27964 raft_consensus.cc:729] T a9dc9360dfa64f1a842b592b86ca8a37 P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d41a707b8bc4282ae3a5f954f607358, State: Initialized, Role: FOLLOWER
I20260807 06:23:02.408947 27964 consensus_queue.cc:260] T a9dc9360dfa64f1a842b592b86ca8a37 P 2d41a707b8bc4282ae3a5f954f607358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.409029 27962 ts_tablet_manager.cc:1428] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 06:23:02.409191 27973 tablet_replica.cc:439] T a9dc9360dfa64f1a842b592b86ca8a37 P 2d41a707b8bc4282ae3a5f954f607358: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } } }
I20260807 06:23:02.409373 27973 tablet_replica.cc:442] T a9dc9360dfa64f1a842b592b86ca8a37 P 2d41a707b8bc4282ae3a5f954f607358: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:23:02.409036 27991 tablet_replica.cc:439] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } } }
I20260807 06:23:02.409797 27991 tablet_replica.cc:442] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:23:02.409209 27964 ts_tablet_manager.cc:1428] T a9dc9360dfa64f1a842b592b86ca8a37 P 2d41a707b8bc4282ae3a5f954f607358: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260807 06:23:02.410770 27963 tablet_bootstrap.cc:492] T a9dc9360dfa64f1a842b592b86ca8a37 P 6a32044d9e4d4bd385dada5ce25fb0a9: Bootstrap starting.
I20260807 06:23:02.411953 27963 tablet_bootstrap.cc:654] T a9dc9360dfa64f1a842b592b86ca8a37 P 6a32044d9e4d4bd385dada5ce25fb0a9: Neither blocks nor log segments found. Creating new log.
I20260807 06:23:02.413445 27963 tablet_bootstrap.cc:492] T a9dc9360dfa64f1a842b592b86ca8a37 P 6a32044d9e4d4bd385dada5ce25fb0a9: No bootstrap required, opened a new log
I20260807 06:23:02.413519 27963 ts_tablet_manager.cc:1397] T a9dc9360dfa64f1a842b592b86ca8a37 P 6a32044d9e4d4bd385dada5ce25fb0a9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 06:23:02.413890 27963 raft_consensus.cc:348] T a9dc9360dfa64f1a842b592b86ca8a37 P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.413987 27963 raft_consensus.cc:374] T a9dc9360dfa64f1a842b592b86ca8a37 P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:23:02.414013 27963 raft_consensus.cc:729] T a9dc9360dfa64f1a842b592b86ca8a37 P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a32044d9e4d4bd385dada5ce25fb0a9, State: Initialized, Role: FOLLOWER
I20260807 06:23:02.414149 27963 consensus_queue.cc:260] T a9dc9360dfa64f1a842b592b86ca8a37 P 6a32044d9e4d4bd385dada5ce25fb0a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.414464 27963 ts_tablet_manager.cc:1428] T a9dc9360dfa64f1a842b592b86ca8a37 P 6a32044d9e4d4bd385dada5ce25fb0a9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260807 06:23:02.414412 27971 tablet_replica.cc:439] T a9dc9360dfa64f1a842b592b86ca8a37 P 6a32044d9e4d4bd385dada5ce25fb0a9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } } }
I20260807 06:23:02.414585 27971 tablet_replica.cc:442] T a9dc9360dfa64f1a842b592b86ca8a37 P 6a32044d9e4d4bd385dada5ce25fb0a9: This TxnStatusTablet replica's current role is: FOLLOWER
W20260807 06:23:02.419586 27919 tablet.cc:2369] T a9dc9360dfa64f1a842b592b86ca8a37 P 6a32044d9e4d4bd385dada5ce25fb0a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 06:23:02.421679 27715 tablet.cc:2369] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:23:02.429623 27991 raft_consensus.cc:482] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:23:02.429785 27991 raft_consensus.cc:504] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.430272 27991 leader_election.cc:288] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d41a707b8bc4282ae3a5f954f607358 (127.15.254.1:46267), 6a32044d9e4d4bd385dada5ce25fb0a9 (127.15.254.3:40955)
I20260807 06:23:02.430753 27429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9dc9360dfa64f1a842b592b86ca8a37" candidate_uuid: "5d3c6d8875714934b60b33af8de46b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d41a707b8bc4282ae3a5f954f607358" is_pre_election: true
I20260807 06:23:02.430765 27855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9dc9360dfa64f1a842b592b86ca8a37" candidate_uuid: "5d3c6d8875714934b60b33af8de46b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" is_pre_election: true
I20260807 06:23:02.430912 27429 raft_consensus.cc:2454] T a9dc9360dfa64f1a842b592b86ca8a37 P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d3c6d8875714934b60b33af8de46b82 in term 0.
I20260807 06:23:02.430927 27855 raft_consensus.cc:2454] T a9dc9360dfa64f1a842b592b86ca8a37 P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d3c6d8875714934b60b33af8de46b82 in term 0.
I20260807 06:23:02.431267 27541 leader_election.cc:302] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d41a707b8bc4282ae3a5f954f607358, 5d3c6d8875714934b60b33af8de46b82; no voters:
I20260807 06:23:02.431492 27991 raft_consensus.cc:2781] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:23:02.431545 27991 raft_consensus.cc:482] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:23:02.431588 27991 raft_consensus.cc:3037] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:23:02.433106 27991 raft_consensus.cc:504] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.433528 27991 leader_election.cc:288] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82 [CANDIDATE]: Term 1 election: Requested vote from peers 2d41a707b8bc4282ae3a5f954f607358 (127.15.254.1:46267), 6a32044d9e4d4bd385dada5ce25fb0a9 (127.15.254.3:40955)
I20260807 06:23:02.433879 27429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9dc9360dfa64f1a842b592b86ca8a37" candidate_uuid: "5d3c6d8875714934b60b33af8de46b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d41a707b8bc4282ae3a5f954f607358"
I20260807 06:23:02.433879 27855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9dc9360dfa64f1a842b592b86ca8a37" candidate_uuid: "5d3c6d8875714934b60b33af8de46b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9"
I20260807 06:23:02.434032 27429 raft_consensus.cc:3037] T a9dc9360dfa64f1a842b592b86ca8a37 P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:23:02.434082 27855 raft_consensus.cc:3037] T a9dc9360dfa64f1a842b592b86ca8a37 P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:23:02.435264 27429 raft_consensus.cc:2454] T a9dc9360dfa64f1a842b592b86ca8a37 P 2d41a707b8bc4282ae3a5f954f607358 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d3c6d8875714934b60b33af8de46b82 in term 1.
I20260807 06:23:02.435264 27855 raft_consensus.cc:2454] T a9dc9360dfa64f1a842b592b86ca8a37 P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d3c6d8875714934b60b33af8de46b82 in term 1.
I20260807 06:23:02.435626 27542 leader_election.cc:302] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d3c6d8875714934b60b33af8de46b82, 6a32044d9e4d4bd385dada5ce25fb0a9; no voters:
I20260807 06:23:02.435829 27991 raft_consensus.cc:2781] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:23:02.436137 27991 raft_consensus.cc:686] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82 [term 1 LEADER]: Becoming Leader. State: Replica: 5d3c6d8875714934b60b33af8de46b82, State: Running, Role: LEADER
I20260807 06:23:02.436326 27991 consensus_queue.cc:237] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.437680 27977 tablet_replica.cc:439] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82: TxnStatusTablet state changed. Reason: New leader 5d3c6d8875714934b60b33af8de46b82. Latest consensus state: current_term: 1 leader_uuid: "5d3c6d8875714934b60b33af8de46b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } } }
I20260807 06:23:02.437799 27977 tablet_replica.cc:442] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82: This TxnStatusTablet replica's current role is: LEADER
I20260807 06:23:02.438115 28025 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 06:23:02.438290 28025 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 06:23:02.438971 27196 catalog_manager.cc:5495] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d3c6d8875714934b60b33af8de46b82 (127.15.254.2). New cstate: current_term: 1 leader_uuid: "5d3c6d8875714934b60b33af8de46b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } health_report { overall_health: UNKNOWN } } }
I20260807 06:23:02.459810 28020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:02.455513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:33975 (local address 127.0.0.1:56906)
0807 06:23:02.455741 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:02.455763 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:02.455968 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:02.456345 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:02.456349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:02.456358 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 06:23:02.456581 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:02.456590 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.457474 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.457479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:02.458178 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:02.458188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.458740 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.458745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:02.458860 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:02.458912 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 06:23:02.459447 (+ 535us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 06:23:02.459468 (+ 21us) client_negotiation.cc:769] Sending connection context
0807 06:23:02.459564 (+ 96us) client_negotiation.cc:240] Negotiation successful
0807 06:23:02.459657 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260807 06:23:02.459787 27955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:02.455725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56906 (local address 127.15.254.2:33975)
0807 06:23:02.455888 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:02.455894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:02.455912 (+ 18us) server_negotiation.cc:407] Connection header received
0807 06:23:02.456138 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:02.456144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:02.456206 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:02.456303 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:23:02.456718 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.457311 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.458360 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.458601 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.458739 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:02.459085 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:23:02.459283 (+ 198us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:23:02.459463 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:02.459591 (+ 128us) server_negotiation.cc:299] Negotiation successful
0807 06:23:02.459657 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 06:23:02.462867 27855 raft_consensus.cc:1261] T a9dc9360dfa64f1a842b592b86ca8a37 P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 1 FOLLOWER]: Refusing update from remote peer 5d3c6d8875714934b60b33af8de46b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:23:02.462888 27429 raft_consensus.cc:1261] T a9dc9360dfa64f1a842b592b86ca8a37 P 2d41a707b8bc4282ae3a5f954f607358 [term 1 FOLLOWER]: Refusing update from remote peer 5d3c6d8875714934b60b33af8de46b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:23:02.463392 27991 consensus_queue.cc:1035] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:23:02.463634 27977 consensus_queue.cc:1035] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:23:02.465235 27973 tablet_replica.cc:439] T a9dc9360dfa64f1a842b592b86ca8a37 P 2d41a707b8bc4282ae3a5f954f607358: TxnStatusTablet state changed. Reason: New leader 5d3c6d8875714934b60b33af8de46b82. Latest consensus state: current_term: 1 leader_uuid: "5d3c6d8875714934b60b33af8de46b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } } }
I20260807 06:23:02.465366 27973 tablet_replica.cc:442] T a9dc9360dfa64f1a842b592b86ca8a37 P 2d41a707b8bc4282ae3a5f954f607358: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:23:02.466523 27971 tablet_replica.cc:439] T a9dc9360dfa64f1a842b592b86ca8a37 P 6a32044d9e4d4bd385dada5ce25fb0a9: TxnStatusTablet state changed. Reason: New leader 5d3c6d8875714934b60b33af8de46b82. Latest consensus state: current_term: 1 leader_uuid: "5d3c6d8875714934b60b33af8de46b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } } }
I20260807 06:23:02.466668 27971 tablet_replica.cc:442] T a9dc9360dfa64f1a842b592b86ca8a37 P 6a32044d9e4d4bd385dada5ce25fb0a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:23:02.468523 27977 tablet_replica.cc:439] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5d3c6d8875714934b60b33af8de46b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } } }
I20260807 06:23:02.468677 27977 tablet_replica.cc:442] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82: This TxnStatusTablet replica's current role is: LEADER
I20260807 06:23:02.468595 27970 tablet_replica.cc:439] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5d3c6d8875714934b60b33af8de46b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } } }
I20260807 06:23:02.469627 27970 tablet_replica.cc:442] T a9dc9360dfa64f1a842b592b86ca8a37 P 5d3c6d8875714934b60b33af8de46b82: This TxnStatusTablet replica's current role is: LEADER
I20260807 06:23:02.470232 27971 tablet_replica.cc:439] T a9dc9360dfa64f1a842b592b86ca8a37 P 6a32044d9e4d4bd385dada5ce25fb0a9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5d3c6d8875714934b60b33af8de46b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } } }
I20260807 06:23:02.470372 27971 tablet_replica.cc:442] T a9dc9360dfa64f1a842b592b86ca8a37 P 6a32044d9e4d4bd385dada5ce25fb0a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:23:02.473026 27973 tablet_replica.cc:439] T a9dc9360dfa64f1a842b592b86ca8a37 P 2d41a707b8bc4282ae3a5f954f607358: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5d3c6d8875714934b60b33af8de46b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } } }
I20260807 06:23:02.473160 27973 tablet_replica.cc:442] T a9dc9360dfa64f1a842b592b86ca8a37 P 2d41a707b8bc4282ae3a5f954f607358: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:23:02.473651 27984 mvcc.cc:205] Tried to move back new op lower bound from 7315799172963028992 to 7315799172861661184. Current Snapshot: MvccSnapshot[applied={T|T < 7315799172963028992 or (T in {7315799172963028992})}]
I20260807 06:23:02.479808 27958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:02.475114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41456 (local address 127.15.254.1:46267)
0807 06:23:02.475267 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:02.475272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:02.475286 (+ 14us) server_negotiation.cc:407] Connection header received
0807 06:23:02.475371 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:02.475374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:02.475440 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:02.475586 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:23:02.475852 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.476601 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.477863 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.478158 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.478254 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:02.478609 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:23:02.479361 (+ 752us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:23:02.479466 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:02.479582 (+ 116us) server_negotiation.cc:299] Negotiation successful
0807 06:23:02.479641 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 06:23:02.492141 28036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:02.488411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:33975 (local address 127.15.254.1:50927)
0807 06:23:02.488766 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:02.488784 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:02.488936 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:02.489226 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:02.489229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:02.489238 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 06:23:02.489383 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:02.489389 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.490146 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.490149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:02.490832 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:02.490841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.491319 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.491323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:02.491431 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:02.491480 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 06:23:02.491808 (+ 328us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 06:23:02.491817 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 06:23:02.491883 (+ 66us) client_negotiation.cc:240] Negotiation successful
0807 06:23:02.491970 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":127,"threads_started":1}
I20260807 06:23:02.492121 27955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:02.488611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:50927 (local address 127.15.254.2:33975)
0807 06:23:02.488749 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:02.488754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:02.488871 (+ 117us) server_negotiation.cc:407] Connection header received
0807 06:23:02.489051 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:02.489054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:02.489121 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:02.489211 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:23:02.489497 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.490023 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.490963 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.491197 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.491311 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:02.491631 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:23:02.491716 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:23:02.491798 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:02.491894 (+ 96us) server_negotiation.cc:299] Negotiation successful
0807 06:23:02.491959 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 06:23:02.500216 27855 raft_consensus.cc:1261] T dba959bf3c72480b8591702901953d5e P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 1 FOLLOWER]: Refusing update from remote peer 2d41a707b8bc4282ae3a5f954f607358: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:23:02.500608 27647 raft_consensus.cc:1261] T dba959bf3c72480b8591702901953d5e P 5d3c6d8875714934b60b33af8de46b82 [term 1 FOLLOWER]: Refusing update from remote peer 2d41a707b8bc4282ae3a5f954f607358: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:23:02.500777 27973 consensus_queue.cc:1035] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:23:02.501320 27973 consensus_queue.cc:1035] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:23:02.524593 28046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:02.519934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:46267 (local address 127.15.254.2:51851)
0807 06:23:02.520555 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:02.520573 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:02.520734 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:02.521149 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:02.521151 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:02.521160 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 06:23:02.521282 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:02.521291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.522211 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.522214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:02.523126 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:02.523141 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.523709 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.523711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:02.523772 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:02.523819 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 06:23:02.524227 (+ 408us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 06:23:02.524232 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 06:23:02.524312 (+ 80us) client_negotiation.cc:240] Negotiation successful
0807 06:23:02.524403 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":143,"threads_started":1}
I20260807 06:23:02.524577 27958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:02.520259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:51851 (local address 127.15.254.1:46267)
0807 06:23:02.520407 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:02.520412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:02.520644 (+ 232us) server_negotiation.cc:407] Connection header received
0807 06:23:02.520863 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:02.520866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:02.520930 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:02.520980 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:23:02.521398 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.522074 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.523304 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.523542 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.523611 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:02.524009 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:23:02.524098 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:23:02.524149 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:02.524345 (+ 196us) server_negotiation.cc:299] Negotiation successful
0807 06:23:02.524403 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 06:23:02.552011 27195 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:23:02.552215 27195 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:23:02.554873 27195 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7: Sending DeleteTablet for 3 replicas of tablet dba959bf3c72480b8591702901953d5e
I20260807 06:23:02.555507 27394 tablet_service.cc:1514] Processing DeleteTablet for tablet dba959bf3c72480b8591702901953d5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:23:02 UTC) from {username='oryx', principal='oryx/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:41454
I20260807 06:23:02.555704 27997 tablet_replica.cc:330] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358: stopping tablet replica
I20260807 06:23:02.555668 27620 tablet_service.cc:1514] Processing DeleteTablet for tablet dba959bf3c72480b8591702901953d5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:23:02 UTC) from {username='oryx', principal='oryx/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:56884
I20260807 06:23:02.555819 27997 raft_consensus.cc:2229] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:23:02.555876 28000 tablet_replica.cc:330] T dba959bf3c72480b8591702901953d5e P 5d3c6d8875714934b60b33af8de46b82: stopping tablet replica
I20260807 06:23:02.555986 28000 raft_consensus.cc:2229] T dba959bf3c72480b8591702901953d5e P 5d3c6d8875714934b60b33af8de46b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:23:02.556095 28000 raft_consensus.cc:2258] T dba959bf3c72480b8591702901953d5e P 5d3c6d8875714934b60b33af8de46b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:23:02.556162 27997 raft_consensus.cc:2258] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:23:02.557063 28000 ts_tablet_manager.cc:1905] T dba959bf3c72480b8591702901953d5e P 5d3c6d8875714934b60b33af8de46b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:23:02.557108 27997 ts_tablet_manager.cc:1905] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:23:02.560029 27827 tablet_service.cc:1514] Processing DeleteTablet for tablet dba959bf3c72480b8591702901953d5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:23:02 UTC) from {username='oryx', principal='oryx/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:45334
I20260807 06:23:02.560283 27998 tablet_replica.cc:330] T dba959bf3c72480b8591702901953d5e P 6a32044d9e4d4bd385dada5ce25fb0a9: stopping tablet replica
I20260807 06:23:02.560422 27998 raft_consensus.cc:2229] T dba959bf3c72480b8591702901953d5e P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:23:02.560624 27998 raft_consensus.cc:2258] T dba959bf3c72480b8591702901953d5e P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:23:02.561331 28000 ts_tablet_manager.cc:1918] T dba959bf3c72480b8591702901953d5e P 5d3c6d8875714934b60b33af8de46b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:23:02.561421 28000 log.cc:1196] T dba959bf3c72480b8591702901953d5e P 5d3c6d8875714934b60b33af8de46b82: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/dba959bf3c72480b8591702901953d5e
I20260807 06:23:02.561553 27998 ts_tablet_manager.cc:1905] T dba959bf3c72480b8591702901953d5e P 6a32044d9e4d4bd385dada5ce25fb0a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:23:02.561707 28000 ts_tablet_manager.cc:1939] T dba959bf3c72480b8591702901953d5e P 5d3c6d8875714934b60b33af8de46b82: Deleting consensus metadata
I20260807 06:23:02.562559 27175 catalog_manager.cc:4841] TS 5d3c6d8875714934b60b33af8de46b82 (127.15.254.2:33975): tablet dba959bf3c72480b8591702901953d5e (table test-table [id=8f8fe5e5dd99456abacdd6cf932160d7]) successfully deleted
I20260807 06:23:02.565303 27998 ts_tablet_manager.cc:1918] T dba959bf3c72480b8591702901953d5e P 6a32044d9e4d4bd385dada5ce25fb0a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:23:02.565411 27998 log.cc:1196] T dba959bf3c72480b8591702901953d5e P 6a32044d9e4d4bd385dada5ce25fb0a9: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/dba959bf3c72480b8591702901953d5e
I20260807 06:23:02.565707 27998 ts_tablet_manager.cc:1939] T dba959bf3c72480b8591702901953d5e P 6a32044d9e4d4bd385dada5ce25fb0a9: Deleting consensus metadata
I20260807 06:23:02.566584 27178 catalog_manager.cc:4841] TS 6a32044d9e4d4bd385dada5ce25fb0a9 (127.15.254.3:40955): tablet dba959bf3c72480b8591702901953d5e (table test-table [id=8f8fe5e5dd99456abacdd6cf932160d7]) successfully deleted
I20260807 06:23:02.567548 27997 ts_tablet_manager.cc:1918] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:23:02.567677 27997 log.cc:1196] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-0/wal/wals/dba959bf3c72480b8591702901953d5e
I20260807 06:23:02.567956 27997 ts_tablet_manager.cc:1939] T dba959bf3c72480b8591702901953d5e P 2d41a707b8bc4282ae3a5f954f607358: Deleting consensus metadata
I20260807 06:23:02.568755 27176 catalog_manager.cc:4841] TS 2d41a707b8bc4282ae3a5f954f607358 (127.15.254.1:46267): tablet dba959bf3c72480b8591702901953d5e (table test-table [id=8f8fe5e5dd99456abacdd6cf932160d7]) successfully deleted
Aug 07 06:23:02 dist-test-slave-44d2 krb5kdc[27133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 06:23:02 dist-test-slave-44d2 krb5kdc[27133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:02 dist-test-slave-44d2 krb5kdc[27133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
W20260807 06:23:02.600337 27265 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:02.591740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59260 (local address 127.15.254.62:33815)
0807 06:23:02.591879 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:02.591884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:02.592186 (+ 302us) server_negotiation.cc:407] Connection header received
0807 06:23:02.592340 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:02.592344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:02.592404 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:02.592524 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:02.593555 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.594400 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.595643 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.596000 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.596114 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:02.600177 (+ 4063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59260: BlockingRecv error: recv got EOF from 127.0.0.1:59260 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 07 06:23:02 dist-test-slave-44d2 krb5kdc[27133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM
I20260807 06:23:02.613680 27265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:02.602500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59272 (local address 127.15.254.62:33815)
0807 06:23:02.602680 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:02.602686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:02.603101 (+ 415us) server_negotiation.cc:407] Connection header received
0807 06:23:02.603261 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:02.603265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:02.603327 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:02.603429 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:02.604250 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.605086 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.606273 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:02.606600 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:02.606704 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:02.609793 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:02.609798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:02.609801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:23:02.609834 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:02.611656 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:02.612170 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:02.612174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:02.612176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:02.612273 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:02.612667 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:02.612696 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:02.612699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:02.612935 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:02.613087 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:02.613348 (+ 261us) server_negotiation.cc:299] Negotiation successful
0807 06:23:02.613487 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 06:23:02.616912 27195 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:23:02.617408 27195 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:23:02.622953 27394 tablet_service.cc:1467] Processing CreateTablet for tablet 3436de71f4994c3c89030c847e64838f (DEFAULT_TABLE table=test-table [id=51c7a71caef648ef8b1f1f53e84c2def]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:23:02.623131 27827 tablet_service.cc:1467] Processing CreateTablet for tablet 3436de71f4994c3c89030c847e64838f (DEFAULT_TABLE table=test-table [id=51c7a71caef648ef8b1f1f53e84c2def]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:23:02.623296 27394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3436de71f4994c3c89030c847e64838f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:23:02.623440 27827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3436de71f4994c3c89030c847e64838f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:23:02.625855 27964 tablet_bootstrap.cc:492] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358: Bootstrap starting.
I20260807 06:23:02.626843 27964 tablet_bootstrap.cc:654] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358: Neither blocks nor log segments found. Creating new log.
I20260807 06:23:02.627542 27963 tablet_bootstrap.cc:492] T 3436de71f4994c3c89030c847e64838f P 6a32044d9e4d4bd385dada5ce25fb0a9: Bootstrap starting.
I20260807 06:23:02.628772 27963 tablet_bootstrap.cc:654] T 3436de71f4994c3c89030c847e64838f P 6a32044d9e4d4bd385dada5ce25fb0a9: Neither blocks nor log segments found. Creating new log.
I20260807 06:23:02.630486 27620 tablet_service.cc:1467] Processing CreateTablet for tablet 3436de71f4994c3c89030c847e64838f (DEFAULT_TABLE table=test-table [id=51c7a71caef648ef8b1f1f53e84c2def]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:23:02.630656 27964 tablet_bootstrap.cc:492] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358: No bootstrap required, opened a new log
I20260807 06:23:02.630746 27964 ts_tablet_manager.cc:1397] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260807 06:23:02.630825 27620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3436de71f4994c3c89030c847e64838f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:23:02.631220 27964 raft_consensus.cc:348] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.631341 27964 raft_consensus.cc:374] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:23:02.631368 27964 raft_consensus.cc:729] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d41a707b8bc4282ae3a5f954f607358, State: Initialized, Role: FOLLOWER
I20260807 06:23:02.631502 27964 consensus_queue.cc:260] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.632119 27963 tablet_bootstrap.cc:492] T 3436de71f4994c3c89030c847e64838f P 6a32044d9e4d4bd385dada5ce25fb0a9: No bootstrap required, opened a new log
I20260807 06:23:02.632206 27963 ts_tablet_manager.cc:1397] T 3436de71f4994c3c89030c847e64838f P 6a32044d9e4d4bd385dada5ce25fb0a9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260807 06:23:02.632738 27963 raft_consensus.cc:348] T 3436de71f4994c3c89030c847e64838f P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.632853 27963 raft_consensus.cc:374] T 3436de71f4994c3c89030c847e64838f P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:23:02.632891 27963 raft_consensus.cc:729] T 3436de71f4994c3c89030c847e64838f P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a32044d9e4d4bd385dada5ce25fb0a9, State: Initialized, Role: FOLLOWER
I20260807 06:23:02.633785 27962 tablet_bootstrap.cc:492] T 3436de71f4994c3c89030c847e64838f P 5d3c6d8875714934b60b33af8de46b82: Bootstrap starting.
I20260807 06:23:02.634233 27964 ts_tablet_manager.cc:1428] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260807 06:23:02.634481 27963 consensus_queue.cc:260] T 3436de71f4994c3c89030c847e64838f P 6a32044d9e4d4bd385dada5ce25fb0a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.634720 27962 tablet_bootstrap.cc:654] T 3436de71f4994c3c89030c847e64838f P 5d3c6d8875714934b60b33af8de46b82: Neither blocks nor log segments found. Creating new log.
I20260807 06:23:02.634817 27963 ts_tablet_manager.cc:1428] T 3436de71f4994c3c89030c847e64838f P 6a32044d9e4d4bd385dada5ce25fb0a9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260807 06:23:02.637456 27962 tablet_bootstrap.cc:492] T 3436de71f4994c3c89030c847e64838f P 5d3c6d8875714934b60b33af8de46b82: No bootstrap required, opened a new log
I20260807 06:23:02.637583 27962 ts_tablet_manager.cc:1397] T 3436de71f4994c3c89030c847e64838f P 5d3c6d8875714934b60b33af8de46b82: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260807 06:23:02.638087 27962 raft_consensus.cc:348] T 3436de71f4994c3c89030c847e64838f P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.638243 27962 raft_consensus.cc:374] T 3436de71f4994c3c89030c847e64838f P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:23:02.638293 27962 raft_consensus.cc:729] T 3436de71f4994c3c89030c847e64838f P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d3c6d8875714934b60b33af8de46b82, State: Initialized, Role: FOLLOWER
I20260807 06:23:02.638448 27962 consensus_queue.cc:260] T 3436de71f4994c3c89030c847e64838f P 5d3c6d8875714934b60b33af8de46b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.638741 27962 ts_tablet_manager.cc:1428] T 3436de71f4994c3c89030c847e64838f P 5d3c6d8875714934b60b33af8de46b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 06:23:02.951180 27973 raft_consensus.cc:482] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:23:02.951372 27973 raft_consensus.cc:504] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.951989 27973 leader_election.cc:288] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d3c6d8875714934b60b33af8de46b82 (127.15.254.2:33975), 6a32044d9e4d4bd385dada5ce25fb0a9 (127.15.254.3:40955)
I20260807 06:23:02.952596 27647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3436de71f4994c3c89030c847e64838f" candidate_uuid: "2d41a707b8bc4282ae3a5f954f607358" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3c6d8875714934b60b33af8de46b82" is_pre_election: true
I20260807 06:23:02.952600 27851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3436de71f4994c3c89030c847e64838f" candidate_uuid: "2d41a707b8bc4282ae3a5f954f607358" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" is_pre_election: true
I20260807 06:23:02.952772 27851 raft_consensus.cc:2454] T 3436de71f4994c3c89030c847e64838f P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d41a707b8bc4282ae3a5f954f607358 in term 0.
I20260807 06:23:02.952772 27647 raft_consensus.cc:2454] T 3436de71f4994c3c89030c847e64838f P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d41a707b8bc4282ae3a5f954f607358 in term 0.
I20260807 06:23:02.953130 27329 leader_election.cc:302] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d41a707b8bc4282ae3a5f954f607358, 5d3c6d8875714934b60b33af8de46b82; no voters:
I20260807 06:23:02.953367 27973 raft_consensus.cc:2781] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:23:02.953437 27973 raft_consensus.cc:482] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:23:02.953470 27973 raft_consensus.cc:3037] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:23:02.954602 27973 raft_consensus.cc:504] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.955107 27973 leader_election.cc:288] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358 [CANDIDATE]: Term 1 election: Requested vote from peers 5d3c6d8875714934b60b33af8de46b82 (127.15.254.2:33975), 6a32044d9e4d4bd385dada5ce25fb0a9 (127.15.254.3:40955)
I20260807 06:23:02.955569 27851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3436de71f4994c3c89030c847e64838f" candidate_uuid: "2d41a707b8bc4282ae3a5f954f607358" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9"
I20260807 06:23:02.955590 27647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3436de71f4994c3c89030c847e64838f" candidate_uuid: "2d41a707b8bc4282ae3a5f954f607358" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3c6d8875714934b60b33af8de46b82"
I20260807 06:23:02.955689 27851 raft_consensus.cc:3037] T 3436de71f4994c3c89030c847e64838f P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:23:02.955799 27647 raft_consensus.cc:3037] T 3436de71f4994c3c89030c847e64838f P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:23:02.956897 27851 raft_consensus.cc:2454] T 3436de71f4994c3c89030c847e64838f P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d41a707b8bc4282ae3a5f954f607358 in term 1.
I20260807 06:23:02.956991 27647 raft_consensus.cc:2454] T 3436de71f4994c3c89030c847e64838f P 5d3c6d8875714934b60b33af8de46b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d41a707b8bc4282ae3a5f954f607358 in term 1.
I20260807 06:23:02.957333 27331 leader_election.cc:302] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d41a707b8bc4282ae3a5f954f607358, 6a32044d9e4d4bd385dada5ce25fb0a9; no voters:
I20260807 06:23:02.957581 27973 raft_consensus.cc:2781] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:23:02.957692 27973 raft_consensus.cc:686] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358 [term 1 LEADER]: Becoming Leader. State: Replica: 2d41a707b8bc4282ae3a5f954f607358, State: Running, Role: LEADER
I20260807 06:23:02.957872 27973 consensus_queue.cc:237] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:02.959806 27195 catalog_manager.cc:5495] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d41a707b8bc4282ae3a5f954f607358 (127.15.254.1). New cstate: current_term: 1 leader_uuid: "2d41a707b8bc4282ae3a5f954f607358" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } health_report { overall_health: UNKNOWN } } }
I20260807 06:23:03.012892 27958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:03.009312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41470 (local address 127.15.254.1:46267)
0807 06:23:03.009471 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:03.009476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:03.009493 (+ 17us) server_negotiation.cc:407] Connection header received
0807 06:23:03.009583 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:03.009586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:03.009645 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:03.009733 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:23:03.010001 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:03.010564 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:03.011485 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:03.011761 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:03.011874 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:03.012267 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:23:03.012350 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:23:03.012436 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:03.012650 (+ 214us) server_negotiation.cc:299] Negotiation successful
0807 06:23:03.012710 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 06:23:03.015272 27851 raft_consensus.cc:1261] T 3436de71f4994c3c89030c847e64838f P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 1 FOLLOWER]: Refusing update from remote peer 2d41a707b8bc4282ae3a5f954f607358: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:23:03.015280 27647 raft_consensus.cc:1261] T 3436de71f4994c3c89030c847e64838f P 5d3c6d8875714934b60b33af8de46b82 [term 1 FOLLOWER]: Refusing update from remote peer 2d41a707b8bc4282ae3a5f954f607358: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:23:03.015825 28044 consensus_queue.cc:1035] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:23:03.015996 27973 consensus_queue.cc:1035] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:23:03.024076 27195 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:23:03.024230 27195 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:23:03.026332 27195 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7: Sending DeleteTablet for 3 replicas of tablet 3436de71f4994c3c89030c847e64838f
I20260807 06:23:03.026983 27394 tablet_service.cc:1514] Processing DeleteTablet for tablet 3436de71f4994c3c89030c847e64838f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:23:03 UTC) from {username='oryx', principal='oryx/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:41454
I20260807 06:23:03.027017 27827 tablet_service.cc:1514] Processing DeleteTablet for tablet 3436de71f4994c3c89030c847e64838f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:23:03 UTC) from {username='oryx', principal='oryx/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:45334
I20260807 06:23:03.027042 27620 tablet_service.cc:1514] Processing DeleteTablet for tablet 3436de71f4994c3c89030c847e64838f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:23:03 UTC) from {username='oryx', principal='oryx/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:56884
I20260807 06:23:03.027194 27997 tablet_replica.cc:330] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358: stopping tablet replica
I20260807 06:23:03.027206 27998 tablet_replica.cc:330] T 3436de71f4994c3c89030c847e64838f P 6a32044d9e4d4bd385dada5ce25fb0a9: stopping tablet replica
I20260807 06:23:03.027206 28000 tablet_replica.cc:330] T 3436de71f4994c3c89030c847e64838f P 5d3c6d8875714934b60b33af8de46b82: stopping tablet replica
I20260807 06:23:03.027319 27998 raft_consensus.cc:2229] T 3436de71f4994c3c89030c847e64838f P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:23:03.027326 28000 raft_consensus.cc:2229] T 3436de71f4994c3c89030c847e64838f P 5d3c6d8875714934b60b33af8de46b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:23:03.027335 27997 raft_consensus.cc:2229] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:23:03.027455 27998 raft_consensus.cc:2258] T 3436de71f4994c3c89030c847e64838f P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:23:03.027456 28000 raft_consensus.cc:2258] T 3436de71f4994c3c89030c847e64838f P 5d3c6d8875714934b60b33af8de46b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:23:03.027554 27997 raft_consensus.cc:2258] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:23:03.028234 28000 ts_tablet_manager.cc:1905] T 3436de71f4994c3c89030c847e64838f P 5d3c6d8875714934b60b33af8de46b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:23:03.028366 27998 ts_tablet_manager.cc:1905] T 3436de71f4994c3c89030c847e64838f P 6a32044d9e4d4bd385dada5ce25fb0a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:23:03.028699 27997 ts_tablet_manager.cc:1905] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:23:03.030422 28000 ts_tablet_manager.cc:1918] T 3436de71f4994c3c89030c847e64838f P 5d3c6d8875714934b60b33af8de46b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:23:03.030525 28000 log.cc:1196] T 3436de71f4994c3c89030c847e64838f P 5d3c6d8875714934b60b33af8de46b82: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/3436de71f4994c3c89030c847e64838f
I20260807 06:23:03.030786 28000 ts_tablet_manager.cc:1939] T 3436de71f4994c3c89030c847e64838f P 5d3c6d8875714934b60b33af8de46b82: Deleting consensus metadata
I20260807 06:23:03.030921 27998 ts_tablet_manager.cc:1918] T 3436de71f4994c3c89030c847e64838f P 6a32044d9e4d4bd385dada5ce25fb0a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:23:03.031020 27998 log.cc:1196] T 3436de71f4994c3c89030c847e64838f P 6a32044d9e4d4bd385dada5ce25fb0a9: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/3436de71f4994c3c89030c847e64838f
I20260807 06:23:03.031175 27997 ts_tablet_manager.cc:1918] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:23:03.031265 27997 log.cc:1196] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-0/wal/wals/3436de71f4994c3c89030c847e64838f
I20260807 06:23:03.031306 27998 ts_tablet_manager.cc:1939] T 3436de71f4994c3c89030c847e64838f P 6a32044d9e4d4bd385dada5ce25fb0a9: Deleting consensus metadata
I20260807 06:23:03.031523 27997 ts_tablet_manager.cc:1939] T 3436de71f4994c3c89030c847e64838f P 2d41a707b8bc4282ae3a5f954f607358: Deleting consensus metadata
I20260807 06:23:03.031719 27175 catalog_manager.cc:4841] TS 5d3c6d8875714934b60b33af8de46b82 (127.15.254.2:33975): tablet 3436de71f4994c3c89030c847e64838f (table test-table [id=51c7a71caef648ef8b1f1f53e84c2def]) successfully deleted
I20260807 06:23:03.032016 27178 catalog_manager.cc:4841] TS 6a32044d9e4d4bd385dada5ce25fb0a9 (127.15.254.3:40955): tablet 3436de71f4994c3c89030c847e64838f (table test-table [id=51c7a71caef648ef8b1f1f53e84c2def]) successfully deleted
I20260807 06:23:03.032558 27195 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 06:23:03.032706 27176 catalog_manager.cc:4841] TS 2d41a707b8bc4282ae3a5f954f607358 (127.15.254.1:46267): tablet 3436de71f4994c3c89030c847e64838f (table test-table [id=51c7a71caef648ef8b1f1f53e84c2def]) successfully deleted
W20260807 06:23:03.033103 27195 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:23:03.038045 27394 tablet_service.cc:1467] Processing CreateTablet for tablet a74d8fb6856249fcb3251ad1586e6efb (DEFAULT_TABLE table=test-table [id=c3dee1e8735d46fca305112b5a98aa09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:23:03.038201 27620 tablet_service.cc:1467] Processing CreateTablet for tablet a74d8fb6856249fcb3251ad1586e6efb (DEFAULT_TABLE table=test-table [id=c3dee1e8735d46fca305112b5a98aa09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:23:03.038204 27827 tablet_service.cc:1467] Processing CreateTablet for tablet a74d8fb6856249fcb3251ad1586e6efb (DEFAULT_TABLE table=test-table [id=c3dee1e8735d46fca305112b5a98aa09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:23:03.038441 27394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a74d8fb6856249fcb3251ad1586e6efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:23:03.038497 27620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a74d8fb6856249fcb3251ad1586e6efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:23:03.038496 27827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a74d8fb6856249fcb3251ad1586e6efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:23:03.041056 27964 tablet_bootstrap.cc:492] T a74d8fb6856249fcb3251ad1586e6efb P 2d41a707b8bc4282ae3a5f954f607358: Bootstrap starting.
I20260807 06:23:03.042140 27964 tablet_bootstrap.cc:654] T a74d8fb6856249fcb3251ad1586e6efb P 2d41a707b8bc4282ae3a5f954f607358: Neither blocks nor log segments found. Creating new log.
I20260807 06:23:03.042778 27963 tablet_bootstrap.cc:492] T a74d8fb6856249fcb3251ad1586e6efb P 6a32044d9e4d4bd385dada5ce25fb0a9: Bootstrap starting.
I20260807 06:23:03.043398 27964 tablet_bootstrap.cc:492] T a74d8fb6856249fcb3251ad1586e6efb P 2d41a707b8bc4282ae3a5f954f607358: No bootstrap required, opened a new log
I20260807 06:23:03.043489 27964 ts_tablet_manager.cc:1397] T a74d8fb6856249fcb3251ad1586e6efb P 2d41a707b8bc4282ae3a5f954f607358: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260807 06:23:03.043874 27963 tablet_bootstrap.cc:654] T a74d8fb6856249fcb3251ad1586e6efb P 6a32044d9e4d4bd385dada5ce25fb0a9: Neither blocks nor log segments found. Creating new log.
I20260807 06:23:03.044013 27964 raft_consensus.cc:348] T a74d8fb6856249fcb3251ad1586e6efb P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:03.044139 27964 raft_consensus.cc:374] T a74d8fb6856249fcb3251ad1586e6efb P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:23:03.044175 27964 raft_consensus.cc:729] T a74d8fb6856249fcb3251ad1586e6efb P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d41a707b8bc4282ae3a5f954f607358, State: Initialized, Role: FOLLOWER
I20260807 06:23:03.044320 27964 consensus_queue.cc:260] T a74d8fb6856249fcb3251ad1586e6efb P 2d41a707b8bc4282ae3a5f954f607358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:03.044579 27962 tablet_bootstrap.cc:492] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82: Bootstrap starting.
I20260807 06:23:03.044653 27964 ts_tablet_manager.cc:1428] T a74d8fb6856249fcb3251ad1586e6efb P 2d41a707b8bc4282ae3a5f954f607358: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260807 06:23:03.045475 27962 tablet_bootstrap.cc:654] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82: Neither blocks nor log segments found. Creating new log.
I20260807 06:23:03.046692 27962 tablet_bootstrap.cc:492] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82: No bootstrap required, opened a new log
I20260807 06:23:03.046692 27963 tablet_bootstrap.cc:492] T a74d8fb6856249fcb3251ad1586e6efb P 6a32044d9e4d4bd385dada5ce25fb0a9: No bootstrap required, opened a new log
I20260807 06:23:03.046782 27963 ts_tablet_manager.cc:1397] T a74d8fb6856249fcb3251ad1586e6efb P 6a32044d9e4d4bd385dada5ce25fb0a9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260807 06:23:03.046782 27962 ts_tablet_manager.cc:1397] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260807 06:23:03.047185 27962 raft_consensus.cc:348] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:03.047231 27963 raft_consensus.cc:348] T a74d8fb6856249fcb3251ad1586e6efb P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:03.047299 27962 raft_consensus.cc:374] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:23:03.047336 27963 raft_consensus.cc:374] T a74d8fb6856249fcb3251ad1586e6efb P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:23:03.047331 27962 raft_consensus.cc:729] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d3c6d8875714934b60b33af8de46b82, State: Initialized, Role: FOLLOWER
I20260807 06:23:03.047370 27963 raft_consensus.cc:729] T a74d8fb6856249fcb3251ad1586e6efb P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a32044d9e4d4bd385dada5ce25fb0a9, State: Initialized, Role: FOLLOWER
I20260807 06:23:03.047488 27963 consensus_queue.cc:260] T a74d8fb6856249fcb3251ad1586e6efb P 6a32044d9e4d4bd385dada5ce25fb0a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:03.047482 27962 consensus_queue.cc:260] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:03.047750 27962 ts_tablet_manager.cc:1428] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 06:23:03.047750 27963 ts_tablet_manager.cc:1428] T a74d8fb6856249fcb3251ad1586e6efb P 6a32044d9e4d4bd385dada5ce25fb0a9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260807 06:23:03.050729 27990 raft_consensus.cc:482] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:23:03.050864 27990 raft_consensus.cc:504] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:03.051229 27990 leader_election.cc:288] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d41a707b8bc4282ae3a5f954f607358 (127.15.254.1:46267), 6a32044d9e4d4bd385dada5ce25fb0a9 (127.15.254.3:40955)
I20260807 06:23:03.051584 27851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74d8fb6856249fcb3251ad1586e6efb" candidate_uuid: "5d3c6d8875714934b60b33af8de46b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" is_pre_election: true
I20260807 06:23:03.051721 27851 raft_consensus.cc:2454] T a74d8fb6856249fcb3251ad1586e6efb P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d3c6d8875714934b60b33af8de46b82 in term 0.
I20260807 06:23:03.052007 27542 leader_election.cc:302] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d3c6d8875714934b60b33af8de46b82, 6a32044d9e4d4bd385dada5ce25fb0a9; no voters:
I20260807 06:23:03.052031 27429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74d8fb6856249fcb3251ad1586e6efb" candidate_uuid: "5d3c6d8875714934b60b33af8de46b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d41a707b8bc4282ae3a5f954f607358" is_pre_election: true
I20260807 06:23:03.052168 27429 raft_consensus.cc:2454] T a74d8fb6856249fcb3251ad1586e6efb P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d3c6d8875714934b60b33af8de46b82 in term 0.
I20260807 06:23:03.052373 27990 raft_consensus.cc:2781] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:23:03.052438 27990 raft_consensus.cc:482] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:23:03.052508 27990 raft_consensus.cc:3037] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:23:03.053653 27990 raft_consensus.cc:504] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:03.053992 27990 leader_election.cc:288] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82 [CANDIDATE]: Term 1 election: Requested vote from peers 2d41a707b8bc4282ae3a5f954f607358 (127.15.254.1:46267), 6a32044d9e4d4bd385dada5ce25fb0a9 (127.15.254.3:40955)
I20260807 06:23:03.054340 27429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74d8fb6856249fcb3251ad1586e6efb" candidate_uuid: "5d3c6d8875714934b60b33af8de46b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d41a707b8bc4282ae3a5f954f607358"
I20260807 06:23:03.054354 27851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74d8fb6856249fcb3251ad1586e6efb" candidate_uuid: "5d3c6d8875714934b60b33af8de46b82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9"
I20260807 06:23:03.054478 27429 raft_consensus.cc:3037] T a74d8fb6856249fcb3251ad1586e6efb P 2d41a707b8bc4282ae3a5f954f607358 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:23:03.054478 27851 raft_consensus.cc:3037] T a74d8fb6856249fcb3251ad1586e6efb P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:23:03.055567 27851 raft_consensus.cc:2454] T a74d8fb6856249fcb3251ad1586e6efb P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d3c6d8875714934b60b33af8de46b82 in term 1.
I20260807 06:23:03.055567 27429 raft_consensus.cc:2454] T a74d8fb6856249fcb3251ad1586e6efb P 2d41a707b8bc4282ae3a5f954f607358 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d3c6d8875714934b60b33af8de46b82 in term 1.
I20260807 06:23:03.055884 27542 leader_election.cc:302] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d3c6d8875714934b60b33af8de46b82, 6a32044d9e4d4bd385dada5ce25fb0a9; no voters:
I20260807 06:23:03.056110 27990 raft_consensus.cc:2781] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:23:03.056268 27990 raft_consensus.cc:686] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82 [term 1 LEADER]: Becoming Leader. State: Replica: 5d3c6d8875714934b60b33af8de46b82, State: Running, Role: LEADER
I20260807 06:23:03.056435 27990 consensus_queue.cc:237] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } }
I20260807 06:23:03.058117 27194 catalog_manager.cc:5495] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d3c6d8875714934b60b33af8de46b82 (127.15.254.2). New cstate: current_term: 1 leader_uuid: "5d3c6d8875714934b60b33af8de46b82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d3c6d8875714934b60b33af8de46b82" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 33975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 } health_report { overall_health: UNKNOWN } } }
I20260807 06:23:03.075526 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:03.071693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56916 (local address 127.15.254.2:33975)
0807 06:23:03.072046 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:03.072054 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:03.072069 (+ 15us) server_negotiation.cc:407] Connection header received
0807 06:23:03.072112 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:03.072115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:03.072166 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:03.072366 (+ 200us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:23:03.072706 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:03.073397 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:03.074300 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:03.074552 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:03.074642 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:03.074914 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:23:03.074999 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:23:03.075058 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:03.075318 (+ 260us) server_negotiation.cc:299] Negotiation successful
0807 06:23:03.075376 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":150,"threads_started":1}
I20260807 06:23:03.087069 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:03.083687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:49565 (local address 127.15.254.2:33975)
0807 06:23:03.083862 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:03.083866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:03.084042 (+ 176us) server_negotiation.cc:407] Connection header received
0807 06:23:03.084127 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:03.084129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:03.084179 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:03.084255 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:23:03.084553 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:03.085111 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:03.085944 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:03.086207 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:03.086344 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:03.086606 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:23:03.086690 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:23:03.086753 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:03.086877 (+ 124us) server_negotiation.cc:299] Negotiation successful
0807 06:23:03.086926 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 06:23:03.087071 28088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:03.083607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:33975 (local address 127.15.254.2:49565)
0807 06:23:03.083911 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:03.083924 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:03.084026 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:03.084284 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:03.084286 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:03.084294 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 06:23:03.084390 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:03.084394 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:03.085212 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:03.085214 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:03.085812 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:03.085819 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:03.086347 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:03.086350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:03.086432 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:03.086461 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 06:23:03.086778 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 06:23:03.086784 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 06:23:03.086841 (+ 57us) client_negotiation.cc:240] Negotiation successful
0807 06:23:03.086919 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":121,"threads_started":1}
I20260807 06:23:03.093986 27429 raft_consensus.cc:1261] T a74d8fb6856249fcb3251ad1586e6efb P 2d41a707b8bc4282ae3a5f954f607358 [term 1 FOLLOWER]: Refusing update from remote peer 5d3c6d8875714934b60b33af8de46b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:23:03.094147 27851 raft_consensus.cc:1261] T a74d8fb6856249fcb3251ad1586e6efb P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 1 FOLLOWER]: Refusing update from remote peer 5d3c6d8875714934b60b33af8de46b82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:23:03.094475 28081 consensus_queue.cc:1035] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d41a707b8bc4282ae3a5f954f607358" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:23:03.094638 27990 consensus_queue.cc:1035] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a32044d9e4d4bd385dada5ce25fb0a9" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 40955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:23:03.132158 27194 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:23:03.132318 27194 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:23:03.134559 27194 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff5b7a8e83ca458c896ba0d82513a6e7: Sending DeleteTablet for 3 replicas of tablet a74d8fb6856249fcb3251ad1586e6efb
I20260807 06:23:03.135200 27394 tablet_service.cc:1514] Processing DeleteTablet for tablet a74d8fb6856249fcb3251ad1586e6efb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:23:03 UTC) from {username='oryx', principal='oryx/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:41454
I20260807 06:23:03.135324 27827 tablet_service.cc:1514] Processing DeleteTablet for tablet a74d8fb6856249fcb3251ad1586e6efb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:23:03 UTC) from {username='oryx', principal='oryx/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:45334
I20260807 06:23:03.135545 27998 tablet_replica.cc:330] T a74d8fb6856249fcb3251ad1586e6efb P 6a32044d9e4d4bd385dada5ce25fb0a9: stopping tablet replica
I20260807 06:23:03.135607 27619 tablet_service.cc:1514] Processing DeleteTablet for tablet a74d8fb6856249fcb3251ad1586e6efb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:23:03 UTC) from {username='oryx', principal='oryx/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:56884
I20260807 06:23:03.135694 27998 raft_consensus.cc:2229] T a74d8fb6856249fcb3251ad1586e6efb P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:23:03.135694 27997 tablet_replica.cc:330] T a74d8fb6856249fcb3251ad1586e6efb P 2d41a707b8bc4282ae3a5f954f607358: stopping tablet replica
I20260807 06:23:03.135782 28000 tablet_replica.cc:330] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82: stopping tablet replica
I20260807 06:23:03.135838 27998 raft_consensus.cc:2258] T a74d8fb6856249fcb3251ad1586e6efb P 6a32044d9e4d4bd385dada5ce25fb0a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:23:03.135846 27997 raft_consensus.cc:2229] T a74d8fb6856249fcb3251ad1586e6efb P 2d41a707b8bc4282ae3a5f954f607358 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:23:03.135978 27997 raft_consensus.cc:2258] T a74d8fb6856249fcb3251ad1586e6efb P 2d41a707b8bc4282ae3a5f954f607358 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:23:03.135996 28000 raft_consensus.cc:2229] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:23:03.136217 28000 raft_consensus.cc:2258] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:23:03.136919 27997 ts_tablet_manager.cc:1905] T a74d8fb6856249fcb3251ad1586e6efb P 2d41a707b8bc4282ae3a5f954f607358: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:23:03.136919 27998 ts_tablet_manager.cc:1905] T a74d8fb6856249fcb3251ad1586e6efb P 6a32044d9e4d4bd385dada5ce25fb0a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:23:03.136997 28000 ts_tablet_manager.cc:1905] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:23:03.139204 28000 ts_tablet_manager.cc:1918] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:23:03.139286 28000 log.cc:1196] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/a74d8fb6856249fcb3251ad1586e6efb
I20260807 06:23:03.139425 27997 ts_tablet_manager.cc:1918] T a74d8fb6856249fcb3251ad1586e6efb P 2d41a707b8bc4282ae3a5f954f607358: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:23:03.139425 27998 ts_tablet_manager.cc:1918] T a74d8fb6856249fcb3251ad1586e6efb P 6a32044d9e4d4bd385dada5ce25fb0a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:23:03.139510 27998 log.cc:1196] T a74d8fb6856249fcb3251ad1586e6efb P 6a32044d9e4d4bd385dada5ce25fb0a9: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/a74d8fb6856249fcb3251ad1586e6efb
I20260807 06:23:03.139520 27997 log.cc:1196] T a74d8fb6856249fcb3251ad1586e6efb P 2d41a707b8bc4282ae3a5f954f607358: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786083738959718-16376-0/minicluster-data/ts-0/wal/wals/a74d8fb6856249fcb3251ad1586e6efb
I20260807 06:23:03.139771 28000 ts_tablet_manager.cc:1939] T a74d8fb6856249fcb3251ad1586e6efb P 5d3c6d8875714934b60b33af8de46b82: Deleting consensus metadata
I20260807 06:23:03.139794 27998 ts_tablet_manager.cc:1939] T a74d8fb6856249fcb3251ad1586e6efb P 6a32044d9e4d4bd385dada5ce25fb0a9: Deleting consensus metadata
I20260807 06:23:03.139984 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 27301
I20260807 06:23:03.140017 27997 ts_tablet_manager.cc:1939] T a74d8fb6856249fcb3251ad1586e6efb P 2d41a707b8bc4282ae3a5f954f607358: Deleting consensus metadata
I20260807 06:23:03.146260 27175 catalog_manager.cc:4841] TS 5d3c6d8875714934b60b33af8de46b82 (127.15.254.2:33975): tablet a74d8fb6856249fcb3251ad1586e6efb (table test-table [id=c3dee1e8735d46fca305112b5a98aa09]) successfully deleted
I20260807 06:23:03.146592 27178 catalog_manager.cc:4841] TS 6a32044d9e4d4bd385dada5ce25fb0a9 (127.15.254.3:40955): tablet a74d8fb6856249fcb3251ad1586e6efb (table test-table [id=c3dee1e8735d46fca305112b5a98aa09]) successfully deleted
W20260807 06:23:03.154701 27176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.254.1:46267 (error 108)
I20260807 06:23:03.154768 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 27509
I20260807 06:23:03.160717 28095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:03.159864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:46267 (local address 127.0.0.1:41476)
0807 06:23:03.160270 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:03.160366 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.1:46267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":211,"threads_started":1}
W20260807 06:23:03.161603 27176 catalog_manager.cc:4584] TS 2d41a707b8bc4282ae3a5f954f607358 (127.15.254.1:46267): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a74d8fb6856249fcb3251ad1586e6efb: Network error: Client connection negotiation failed: client connection to 127.15.254.1:46267: connect: Connection refused (error 111)
I20260807 06:23:03.174789 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 27730
I20260807 06:23:03.190482 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 27152
2026-08-07T06:23:03Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3701 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:23:03 dist-test-slave-44d2 krb5kdc[28099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:23:03 dist-test-slave-44d2 krb5kdc[28099](info): set up 2 sockets
Aug 07 06:23:03 dist-test-slave-44d2 krb5kdc[28099](info): commencing operation
krb5kdc: starting...
W20260807 06:23:04.446362 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.221s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:23:04 dist-test-slave-44d2 krb5kdc[28099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:23:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:23:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:23:04.584393 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:36437
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:36437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:23:04.705106 28119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:23:04.705379 28119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:23:04.705420 28119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:23:04.709431 28119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:23:04.709494 28119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:23:04.709513 28119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:23:04.709530 28119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:23:04.709545 28119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:23:04.714881 28119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:36437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:36437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:23:04.715899 28119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:23:04.716965 28119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:23:04.726629 28129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:04.726816 28119 server_base.cc:1034] running on GCE node
W20260807 06:23:04.726629 28130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:04.726701 28132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:04.727640 28119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:23:04.728915 28119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:23:04.730144 28119 hybrid_clock.cc:648] HybridClock initialized: now 1786083784730132 us; error 42 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-07T06:23:04Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1540 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:23:04 dist-test-slave-44d2 krb5kdc[28140](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:23:04 dist-test-slave-44d2 krb5kdc[28140](info): set up 2 sockets
Aug 07 06:23:04 dist-test-slave-44d2 krb5kdc[28140](info): commencing operation
krb5kdc: starting...
W20260807 06:23:05.972658 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.207s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:23:06 dist-test-slave-44d2 krb5kdc[28140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:23:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:23:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:23:06.111718 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:34449
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:34449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:23:06.231631 28159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:23:06.231958 28159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:23:06.232059 28159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:23:06.236101 28159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:23:06.236197 28159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:23:06.236235 28159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:23:06.236272 28159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:23:06.236305 28159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:23:06.241654 28159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:34449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:34449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:23:06.242889 28159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:23:06.243969 28159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:23:06.253149 28169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:06.253168 28166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:06.253255 28167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:06.253567 28159 server_base.cc:1034] running on GCE node
I20260807 06:23:06.254252 28159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:23:06.255435 28159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:23:06.256620 28159 hybrid_clock.cc:648] HybridClock initialized: now 1786083786256600 us; error 30 us; skew 500 ppm
Aug 07 06:23:06 dist-test-slave-44d2 krb5kdc[28140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083786, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:23:06.260324 28159 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:23:06.261349 28159 webserver.cc:460] Webserver started at http://127.15.254.62:41689/ using document root <none> and password file <none>
I20260807 06:23:06.262022 28159 fs_manager.cc:359] Metadata directory not provided
I20260807 06:23:06.262123 28159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:23:06.262351 28159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:23:06.264631 28159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "1c667c1ba8854e77bd13e4e958a0c12b"
format_stamp: "Formatted at 2026-08-07 06:23:06 on dist-test-slave-44d2"
server_key: "8c949d4a6df494edcb59e1297bb184eb"
server_key_iv: "7fc92d9fcc8bec8735ca1caeb6dc402a"
server_key_version: "encryptionkey@0"
I20260807 06:23:06.265244 28159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "1c667c1ba8854e77bd13e4e958a0c12b"
format_stamp: "Formatted at 2026-08-07 06:23:06 on dist-test-slave-44d2"
server_key: "8c949d4a6df494edcb59e1297bb184eb"
server_key_iv: "7fc92d9fcc8bec8735ca1caeb6dc402a"
server_key_version: "encryptionkey@0"
I20260807 06:23:06.269239 28159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 06:23:06.271828 28181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:06.273257 28159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:23:06.273412 28159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "1c667c1ba8854e77bd13e4e958a0c12b"
format_stamp: "Formatted at 2026-08-07 06:23:06 on dist-test-slave-44d2"
server_key: "8c949d4a6df494edcb59e1297bb184eb"
server_key_iv: "7fc92d9fcc8bec8735ca1caeb6dc402a"
server_key_version: "encryptionkey@0"
I20260807 06:23:06.273530 28159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:23:06.286165 28159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:23:06.289760 28159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:23:06.289994 28159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:23:06.297585 28159 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:34449
I20260807 06:23:06.297650 28247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:34449 every 8 connection(s)
I20260807 06:23:06.298609 28159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:23:06.299533 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 28159
I20260807 06:23:06.299685 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:23:06.300057 16376 external_mini_cluster.cc:1428] Setting key a6beb76047debec7e173cb03519baec1
I20260807 06:23:06.302224 28248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:23:06.309489 28248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b: Bootstrap starting.
Aug 07 06:23:06 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:23:06.311919 28248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b: Neither blocks nor log segments found. Creating new log.
I20260807 06:23:06.312836 28248 log.cc:824] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b: Log is configured to *not* fsync() on all Append() calls
I20260807 06:23:06.315447 28248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b: No bootstrap required, opened a new log
I20260807 06:23:06.316082 28252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:06.301544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46326 (local address 127.15.254.62:34449)
0807 06:23:06.301973 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:06.301999 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:06.302051 (+ 52us) server_negotiation.cc:407] Connection header received
0807 06:23:06.302871 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:06.302899 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:06.303246 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:06.303543 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:06.304612 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:06.305704 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:06.306911 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:06.307411 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:06.307560 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:06.310716 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:06.310721 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:06.310735 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:23:06.310768 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:06.312910 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:06.313510 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:06.313519 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:06.313529 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:06.313656 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:06.313968 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:06.313972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:06.313974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:06.314398 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:06.314636 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:06.314888 (+ 252us) server_negotiation.cc:299] Negotiation successful
0807 06:23:06.315217 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":150,"threads_started":1}
I20260807 06:23:06.319973 28248 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c667c1ba8854e77bd13e4e958a0c12b" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 34449 } }
I20260807 06:23:06.320212 28248 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:23:06.320289 28248 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c667c1ba8854e77bd13e4e958a0c12b, State: Initialized, Role: FOLLOWER
I20260807 06:23:06.320772 28248 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c667c1ba8854e77bd13e4e958a0c12b" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 34449 } }
I20260807 06:23:06.320926 28248 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:23:06.321008 28248 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:23:06.321122 28248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:23:06.322333 28248 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c667c1ba8854e77bd13e4e958a0c12b" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 34449 } }
I20260807 06:23:06.322701 28248 leader_election.cc:302] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c667c1ba8854e77bd13e4e958a0c12b; no voters:
I20260807 06:23:06.323088 28248 leader_election.cc:288] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:23:06.323244 28255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:23:06.323525 28255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [term 1 LEADER]: Becoming Leader. State: Replica: 1c667c1ba8854e77bd13e4e958a0c12b, State: Running, Role: LEADER
I20260807 06:23:06.323839 28255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c667c1ba8854e77bd13e4e958a0c12b" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 34449 } }
I20260807 06:23:06.324410 28248 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:23:06.325613 28256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c667c1ba8854e77bd13e4e958a0c12b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c667c1ba8854e77bd13e4e958a0c12b" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 34449 } } }
I20260807 06:23:06.325762 28256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [sys.catalog]: This master's current role is: LEADER
I20260807 06:23:06.326071 28258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c667c1ba8854e77bd13e4e958a0c12b. Latest consensus state: current_term: 1 leader_uuid: "1c667c1ba8854e77bd13e4e958a0c12b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c667c1ba8854e77bd13e4e958a0c12b" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 34449 } } }
I20260807 06:23:06.326171 28258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [sys.catalog]: This master's current role is: LEADER
I20260807 06:23:06.326354 28269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:23:06.329766 28269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:23:06.335527 28269 catalog_manager.cc:1347] Generated new cluster ID: dda37ff81a5548f391a82d9ef78a9601
I20260807 06:23:06.335603 28269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:23:06.345368 28269 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:23:06.346596 28269 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:23:06.364097 28269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b: Generated new TSK 0
I20260807 06:23:06.365020 28269 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
I20260807 06:23:06.407799 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:0
--local_ip_for_outbound_sockets=127.15.254.1
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:34449
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:23:06.526422 28286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:23:06.526703 28286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:23:06.526785 28286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:23:06.530994 28286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:23:06.531091 28286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:23:06.531204 28286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:23:06.536769 28286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:34449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:23:06.537860 28286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:23:06.538925 28286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:23:06.548825 28295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:06.548815 28294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:06.548815 28298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:06.549238 28286 server_base.cc:1034] running on GCE node
I20260807 06:23:06.549633 28286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:23:06.550323 28286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:23:06.551512 28286 hybrid_clock.cc:648] HybridClock initialized: now 1786083786551499 us; error 28 us; skew 500 ppm
Aug 07 06:23:06 dist-test-slave-44d2 krb5kdc[28140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083786, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:23:06.554452 28286 init.cc:377] Logged in from keytab as kudu/127.15.254.1@KRBTEST.COM (short username kudu)
I20260807 06:23:06.555367 28286 webserver.cc:460] Webserver started at http://127.15.254.1:41105/ using document root <none> and password file <none>
I20260807 06:23:06.555939 28286 fs_manager.cc:359] Metadata directory not provided
I20260807 06:23:06.556035 28286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:23:06.556303 28286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:23:06.558432 28286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/data/instance:
uuid: "10d001851f4c48cba68418e6537fbd98"
format_stamp: "Formatted at 2026-08-07 06:23:06 on dist-test-slave-44d2"
server_key: "d08d0c5d858793a8d135ed702fac7fba"
server_key_iv: "fb857571ed9fb1ca89067accbf708781"
server_key_version: "encryptionkey@0"
I20260807 06:23:06.559012 28286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance:
uuid: "10d001851f4c48cba68418e6537fbd98"
format_stamp: "Formatted at 2026-08-07 06:23:06 on dist-test-slave-44d2"
server_key: "d08d0c5d858793a8d135ed702fac7fba"
server_key_iv: "fb857571ed9fb1ca89067accbf708781"
server_key_version: "encryptionkey@0"
I20260807 06:23:06.562775 28286 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260807 06:23:06.565333 28309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:06.566988 28286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 06:23:06.567163 28286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "10d001851f4c48cba68418e6537fbd98"
format_stamp: "Formatted at 2026-08-07 06:23:06 on dist-test-slave-44d2"
server_key: "d08d0c5d858793a8d135ed702fac7fba"
server_key_iv: "fb857571ed9fb1ca89067accbf708781"
server_key_version: "encryptionkey@0"
I20260807 06:23:06.567301 28286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:23:06.597113 28286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:23:06.600713 28286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:23:06.600939 28286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:23:06.601599 28286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:23:06.602502 28286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:23:06.602587 28286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:06.602664 28286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:23:06.602715 28286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:06.612707 28286 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:46683
I20260807 06:23:06.612776 28478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:46683 every 8 connection(s)
I20260807 06:23:06.613714 28286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:23:06.615470 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 28286
I20260807 06:23:06.615625 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance
I20260807 06:23:06.616060 16376 external_mini_cluster.cc:1428] Setting key faa72677afadb982fb1fc75a05865590
Aug 07 06:23:06 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083786, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.254.2@KRBTEST.COM; defaulting to no policy
I20260807 06:23:06.629765 28252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:06.615686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:36767 (local address 127.15.254.62:34449)
0807 06:23:06.615902 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:06.615908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:06.616928 (+ 1020us) server_negotiation.cc:407] Connection header received
0807 06:23:06.618673 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:06.618676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:06.618720 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:06.618792 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:06.620396 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:06.621375 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:06.622614 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:06.622863 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:06.622990 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:06.625769 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:06.625775 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:06.625783 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:23:06.625807 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:06.627464 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:06.628014 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:06.628022 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:06.628023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:06.628098 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:06.628593 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:06.628597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:06.628598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:06.628786 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:06.628923 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:06.629452 (+ 529us) server_negotiation.cc:299] Negotiation successful
0807 06:23:06.629589 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.2@KRBTEST.COM" created.
I20260807 06:23:06.630715 28482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:06.616179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.1:36767)
0807 06:23:06.616780 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:06.616839 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:06.618439 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:06.618983 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:06.618996 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:06.619558 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:06.620236 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:06.620252 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:06.621539 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:06.621542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:06.622479 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:06.622488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:06.623041 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:06.623046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:06.623217 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:06.623857 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:06.623874 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:06.625595 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:23:06.627587 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:23:06.627594 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:23:06.627609 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:23:06.627898 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:23:06.628231 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:23:06.628234 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:23:06.628236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:23:06.628482 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:23:06.628937 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:23:06.628946 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:23:06.629192 (+ 246us) client_negotiation.cc:769] Sending connection context
0807 06:23:06.629381 (+ 189us) client_negotiation.cc:240] Negotiation successful
0807 06:23:06.629789 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":147,"threads_started":1}
I20260807 06:23:06.633275 28479 heartbeater.cc:347] Connected to a master server at 127.15.254.62:34449
I20260807 06:23:06.633548 28479 heartbeater.cc:464] Registering TS with master...
I20260807 06:23:06.634194 28479 heartbeater.cc:511] Master 127.15.254.62:34449 requested a full tablet report, sending...
I20260807 06:23:06.635900 28201 ts_manager.cc:194] Registered new tserver with Master: 10d001851f4c48cba68418e6537fbd98 (127.15.254.1:46683)
I20260807 06:23:06.637147 28201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.1@KRBTEST.COM'} at 127.15.254.1:36767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
I20260807 06:23:06.649094 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:0
--local_ip_for_outbound_sockets=127.15.254.2
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:34449
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:23:06.774554 28488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:23:06.774809 28488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:23:06.774852 28488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:23:06.779027 28488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:23:06.779083 28488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:23:06.779170 28488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:23:06.784248 28488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:34449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:23:06.785297 28488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:23:06.786334 28488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:23:06.796298 28497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:06.796362 28494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:06.796377 28488 server_base.cc:1034] running on GCE node
W20260807 06:23:06.796285 28495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:06.796983 28488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:23:06.797626 28488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:23:06.798791 28488 hybrid_clock.cc:648] HybridClock initialized: now 1786083786798753 us; error 52 us; skew 500 ppm
Aug 07 06:23:06 dist-test-slave-44d2 krb5kdc[28140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083786, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:23:06.801627 28488 init.cc:377] Logged in from keytab as kudu/127.15.254.2@KRBTEST.COM (short username kudu)
I20260807 06:23:06.802531 28488 webserver.cc:460] Webserver started at http://127.15.254.2:38985/ using document root <none> and password file <none>
I20260807 06:23:06.803135 28488 fs_manager.cc:359] Metadata directory not provided
I20260807 06:23:06.803236 28488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:23:06.803484 28488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:23:06.805723 28488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/data/instance:
uuid: "ffdc6a236f24415bb138f4890b6d12f0"
format_stamp: "Formatted at 2026-08-07 06:23:06 on dist-test-slave-44d2"
server_key: "3a3e7782085afe4682a3581890c024a9"
server_key_iv: "e3907664b45aaedcf45a58665b69e6a7"
server_key_version: "encryptionkey@0"
I20260807 06:23:06.806331 28488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance:
uuid: "ffdc6a236f24415bb138f4890b6d12f0"
format_stamp: "Formatted at 2026-08-07 06:23:06 on dist-test-slave-44d2"
server_key: "3a3e7782085afe4682a3581890c024a9"
server_key_iv: "e3907664b45aaedcf45a58665b69e6a7"
server_key_version: "encryptionkey@0"
I20260807 06:23:06.810281 28488 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:23:06.812860 28511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:06.814159 28488 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 06:23:06.814301 28488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "ffdc6a236f24415bb138f4890b6d12f0"
format_stamp: "Formatted at 2026-08-07 06:23:06 on dist-test-slave-44d2"
server_key: "3a3e7782085afe4682a3581890c024a9"
server_key_iv: "e3907664b45aaedcf45a58665b69e6a7"
server_key_version: "encryptionkey@0"
I20260807 06:23:06.814415 28488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:23:06.839591 28488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:23:06.842986 28488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:23:06.843226 28488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:23:06.843880 28488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:23:06.844880 28488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:23:06.844956 28488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:06.845032 28488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:23:06.845093 28488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:06.854481 28488 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:42111
I20260807 06:23:06.854539 28672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:42111 every 8 connection(s)
I20260807 06:23:06.855454 28488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:23:06.858008 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 28488
I20260807 06:23:06.858156 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance
I20260807 06:23:06.858515 16376 external_mini_cluster.cc:1428] Setting key 10145da82270d46ca8897232baea0e83
Aug 07 06:23:06 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083786, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.254.3@KRBTEST.COM; defaulting to no policy
I20260807 06:23:06.871752 28252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:06.857457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:46169 (local address 127.15.254.62:34449)
0807 06:23:06.857608 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:06.857612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:06.858494 (+ 882us) server_negotiation.cc:407] Connection header received
0807 06:23:06.860197 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:06.860201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:06.860255 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:06.860349 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:06.861588 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:06.862434 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:06.864302 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:06.864758 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:06.864898 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:06.867745 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:06.867749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:06.867752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:23:06.867777 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:06.869368 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:06.869942 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:06.869946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:06.869948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:06.870032 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:06.870405 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:06.870410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:06.870411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:06.870653 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:06.870768 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:06.871393 (+ 625us) server_negotiation.cc:299] Negotiation successful
0807 06:23:06.871536 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 06:23:06.872776 28676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:06.857818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.2:46169)
0807 06:23:06.858312 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:06.858367 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:06.859940 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:06.860554 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:06.860579 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:06.860944 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:06.861433 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:06.861447 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:06.862586 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:06.862592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:06.864124 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:06.864135 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:06.864959 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:06.864963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:06.865119 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:06.865840 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:06.865930 (+ 90us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:06.867573 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:23:06.869499 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:23:06.869506 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:23:06.869522 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:23:06.869816 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:23:06.870138 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:23:06.870141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:23:06.870143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:23:06.870298 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:23:06.870768 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:23:06.870777 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:23:06.871113 (+ 336us) client_negotiation.cc:769] Sending connection context
0807 06:23:06.871338 (+ 225us) client_negotiation.cc:240] Negotiation successful
0807 06:23:06.871703 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":148,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.3@KRBTEST.COM" created.
I20260807 06:23:06.874150 28674 heartbeater.cc:347] Connected to a master server at 127.15.254.62:34449
I20260807 06:23:06.874455 28674 heartbeater.cc:464] Registering TS with master...
I20260807 06:23:06.875015 28674 heartbeater.cc:511] Master 127.15.254.62:34449 requested a full tablet report, sending...
I20260807 06:23:06.876227 28201 ts_manager.cc:194] Registered new tserver with Master: ffdc6a236f24415bb138f4890b6d12f0 (127.15.254.2:42111)
I20260807 06:23:06.876971 28201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.2@KRBTEST.COM'} at 127.15.254.2:46169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
I20260807 06:23:06.891925 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:0
--local_ip_for_outbound_sockets=127.15.254.3
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:34449
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:23:07.010249 28680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:23:07.010536 28680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:23:07.010582 28680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:23:07.014742 28680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:23:07.014813 28680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:23:07.014900 28680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:23:07.020401 28680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:34449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:23:07.021478 28680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:23:07.022735 28680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:23:07.032996 28690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:07.033070 28691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:07.033135 28680 server_base.cc:1034] running on GCE node
W20260807 06:23:07.033018 28693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:07.033706 28680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:23:07.034361 28680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:23:07.035560 28680 hybrid_clock.cc:648] HybridClock initialized: now 1786083787035547 us; error 26 us; skew 500 ppm
Aug 07 06:23:07 dist-test-slave-44d2 krb5kdc[28140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083787, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:23:07.038499 28680 init.cc:377] Logged in from keytab as kudu/127.15.254.3@KRBTEST.COM (short username kudu)
I20260807 06:23:07.039429 28680 webserver.cc:460] Webserver started at http://127.15.254.3:34987/ using document root <none> and password file <none>
I20260807 06:23:07.040026 28680 fs_manager.cc:359] Metadata directory not provided
I20260807 06:23:07.040119 28680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:23:07.040362 28680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:23:07.042670 28680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/data/instance:
uuid: "6c0855cf31d543cb93578c3dcde059f4"
format_stamp: "Formatted at 2026-08-07 06:23:07 on dist-test-slave-44d2"
server_key: "1229dd808e68f0df26f4f8c6638e2ddc"
server_key_iv: "29b9dcc53fc87575aaf9ae9e55fcbfe6"
server_key_version: "encryptionkey@0"
I20260807 06:23:07.043296 28680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance:
uuid: "6c0855cf31d543cb93578c3dcde059f4"
format_stamp: "Formatted at 2026-08-07 06:23:07 on dist-test-slave-44d2"
server_key: "1229dd808e68f0df26f4f8c6638e2ddc"
server_key_iv: "29b9dcc53fc87575aaf9ae9e55fcbfe6"
server_key_version: "encryptionkey@0"
I20260807 06:23:07.047266 28680 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 06:23:07.050092 28704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:07.051487 28680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 06:23:07.051661 28680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "6c0855cf31d543cb93578c3dcde059f4"
format_stamp: "Formatted at 2026-08-07 06:23:07 on dist-test-slave-44d2"
server_key: "1229dd808e68f0df26f4f8c6638e2ddc"
server_key_iv: "29b9dcc53fc87575aaf9ae9e55fcbfe6"
server_key_version: "encryptionkey@0"
I20260807 06:23:07.051775 28680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:23:07.082737 28680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:23:07.086244 28680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:23:07.086470 28680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:23:07.087148 28680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:23:07.088076 28680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:23:07.088124 28680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:07.088191 28680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:23:07.088224 28680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:07.097414 28680 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:43809
I20260807 06:23:07.097442 28867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:43809 every 8 connection(s)
I20260807 06:23:07.098301 28680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:23:07.099732 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 28680
I20260807 06:23:07.099907 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance
I20260807 06:23:07.100265 16376 external_mini_cluster.cc:1428] Setting key 3803f7aaa442daf50cded2ec49a407f6
Aug 07 06:23:07 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083787, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:23:07.112423 28252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:07.100361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:59103 (local address 127.15.254.62:34449)
0807 06:23:07.100548 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:07.100554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:07.101596 (+ 1042us) server_negotiation.cc:407] Connection header received
0807 06:23:07.103139 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:07.103142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:07.103197 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:07.103276 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:07.104439 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.105016 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.106193 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.106511 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.106635 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:07.109028 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:07.109031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:07.109033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:23:07.109058 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:07.110627 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:07.111094 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:07.111097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:07.111098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:07.111173 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:07.111475 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:07.111477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:07.111478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:07.111647 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:07.111723 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:07.112204 (+ 481us) server_negotiation.cc:299] Negotiation successful
0807 06:23:07.112302 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 06:23:07.113440 28870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:07.100863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.3:59103)
0807 06:23:07.101448 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:07.101506 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:07.102914 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:07.103457 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:07.103463 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:07.103810 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:07.104293 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:07.104307 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.105154 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.105158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:07.106052 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:07.106071 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.106679 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.106684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:07.106861 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:07.107386 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:07.107402 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:07.108860 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:23:07.110732 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:23:07.110736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:23:07.110748 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:23:07.110993 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:23:07.111257 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:23:07.111259 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:23:07.111260 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:23:07.111394 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:23:07.111738 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:23:07.111743 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:23:07.111978 (+ 235us) client_negotiation.cc:769] Sending connection context
0807 06:23:07.112135 (+ 157us) client_negotiation.cc:240] Negotiation successful
0807 06:23:07.112391 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":154,"threads_started":1}
I20260807 06:23:07.114596 28868 heartbeater.cc:347] Connected to a master server at 127.15.254.62:34449
I20260807 06:23:07.114843 28868 heartbeater.cc:464] Registering TS with master...
I20260807 06:23:07.115331 28868 heartbeater.cc:511] Master 127.15.254.62:34449 requested a full tablet report, sending...
I20260807 06:23:07.116428 28200 ts_manager.cc:194] Registered new tserver with Master: 6c0855cf31d543cb93578c3dcde059f4 (127.15.254.3:43809)
I20260807 06:23:07.117106 28200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.3@KRBTEST.COM'} at 127.15.254.3:59103
I20260807 06:23:07.126005 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.15.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.1.keytab.
Entry for principal oryx/127.15.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.1.keytab.
WARNING: no policy specified for oryx/127.15.254.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.254.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.2.keytab.
Entry for principal oryx/127.15.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.2.keytab.
WARNING: no policy specified for oryx/127.15.254.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.3.keytab.
Entry for principal oryx/127.15.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.3.keytab.
I20260807 06:23:07.212397 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 28286
I20260807 06:23:07.225471 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 28488
I20260807 06:23:07.243626 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 28680
I20260807 06:23:07.257454 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 28159
I20260807 06:23:07.266986 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:34449
--webserver_interface=127.15.254.62
--webserver_port=41689
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:34449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:23:07.397073 28878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:23:07.397399 28878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:23:07.397478 28878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:23:07.401453 28878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:23:07.401558 28878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:23:07.401597 28878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:23:07.401636 28878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:23:07.401674 28878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:23:07.406945 28878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:34449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:34449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=41689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:23:07.408051 28878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:23:07.409096 28878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:23:07.417819 28885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:07.417940 28890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:07.417953 28878 server_base.cc:1034] running on GCE node
W20260807 06:23:07.417819 28886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:07.418677 28878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:23:07.419742 28878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:23:07.420984 28878 hybrid_clock.cc:648] HybridClock initialized: now 1786083787420968 us; error 26 us; skew 500 ppm
Aug 07 06:23:07 dist-test-slave-44d2 krb5kdc[28140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083787, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:23:07.423882 28878 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:23:07.424937 28878 webserver.cc:460] Webserver started at http://127.15.254.62:41689/ using document root <none> and password file <none>
I20260807 06:23:07.425505 28878 fs_manager.cc:359] Metadata directory not provided
I20260807 06:23:07.425621 28878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:23:07.429535 28878 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:23:07.431715 28904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:07.433236 28878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:23:07.433408 28878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "1c667c1ba8854e77bd13e4e958a0c12b"
format_stamp: "Formatted at 2026-08-07 06:23:06 on dist-test-slave-44d2"
server_key: "8c949d4a6df494edcb59e1297bb184eb"
server_key_iv: "7fc92d9fcc8bec8735ca1caeb6dc402a"
server_key_version: "encryptionkey@0"
I20260807 06:23:07.433995 28878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:23:07.461988 28878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:23:07.465660 28878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:23:07.465888 28878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:23:07.473758 28878 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:34449
I20260807 06:23:07.473788 28975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:34449 every 8 connection(s)
I20260807 06:23:07.474973 28878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:23:07.479432 28976 sys_catalog.cc:263] Verifying existing consensus state
I20260807 06:23:07.481112 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 28878
I20260807 06:23:07.481290 28976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b: Bootstrap starting.
I20260807 06:23:07.481681 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:46683
--local_ip_for_outbound_sockets=127.15.254.1
--tserver_master_addrs=127.15.254.62:34449
--webserver_port=41105
--webserver_interface=127.15.254.1
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.254.1
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:23:07.500037 28976 log.cc:824] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b: Log is configured to *not* fsync() on all Append() calls
I20260807 06:23:07.507005 28976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 06:23:07.507412 28976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b: Bootstrap complete.
I20260807 06:23:07.511921 28976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c667c1ba8854e77bd13e4e958a0c12b" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 34449 } }
I20260807 06:23:07.512364 28976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c667c1ba8854e77bd13e4e958a0c12b, State: Initialized, Role: FOLLOWER
I20260807 06:23:07.512977 28976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c667c1ba8854e77bd13e4e958a0c12b" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 34449 } }
I20260807 06:23:07.513140 28976 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:23:07.513229 28976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:23:07.513346 28976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [term 1 FOLLOWER]: Advancing to term 2
I20260807 06:23:07.515329 28976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c667c1ba8854e77bd13e4e958a0c12b" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 34449 } }
I20260807 06:23:07.515679 28976 leader_election.cc:302] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c667c1ba8854e77bd13e4e958a0c12b; no voters:
I20260807 06:23:07.516002 28976 leader_election.cc:288] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 06:23:07.516207 28981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [term 2 FOLLOWER]: Leader election won for term 2
I20260807 06:23:07.516556 28981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [term 2 LEADER]: Becoming Leader. State: Replica: 1c667c1ba8854e77bd13e4e958a0c12b, State: Running, Role: LEADER
I20260807 06:23:07.516927 28981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c667c1ba8854e77bd13e4e958a0c12b" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 34449 } }
I20260807 06:23:07.517365 28976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:23:07.518569 28982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1c667c1ba8854e77bd13e4e958a0c12b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c667c1ba8854e77bd13e4e958a0c12b" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 34449 } } }
I20260807 06:23:07.518738 28982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [sys.catalog]: This master's current role is: LEADER
I20260807 06:23:07.519488 28983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c667c1ba8854e77bd13e4e958a0c12b. Latest consensus state: current_term: 2 leader_uuid: "1c667c1ba8854e77bd13e4e958a0c12b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c667c1ba8854e77bd13e4e958a0c12b" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 34449 } } }
I20260807 06:23:07.519613 28983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b [sys.catalog]: This master's current role is: LEADER
I20260807 06:23:07.522401 29008 catalog_manager.cc:1259] Loaded cluster ID: dda37ff81a5548f391a82d9ef78a9601
I20260807 06:23:07.522490 29008 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b: loading cluster ID for follower catalog manager: success
I20260807 06:23:07.524283 29008 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b: acquiring CA information for follower catalog manager: success
I20260807 06:23:07.525501 29008 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 06:23:07.525641 28997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:23:07.526340 28997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:23:07.526777 28997 catalog_manager.cc:1259] Loaded cluster ID: dda37ff81a5548f391a82d9ef78a9601
I20260807 06:23:07.526854 28997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:23:07.527477 28997 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:23:07.527897 28997 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1c667c1ba8854e77bd13e4e958a0c12b: Loaded TSK: 0
I20260807 06:23:07.528592 28997 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 06:23:07.614642 28979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:23:07.614957 28979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:23:07.615056 28979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:23:07.619575 28979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:23:07.619652 28979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:23:07.619758 28979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:23:07.625226 28979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.1.keytab
--principal=oryx/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:46683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=41105
--tserver_master_addrs=127.15.254.62:34449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:23:07.626307 28979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:23:07.627379 28979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:23:07.638846 29021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:07.638824 29015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:07.638872 29019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:07.639400 28979 server_base.cc:1034] running on GCE node
I20260807 06:23:07.639882 28979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:23:07.640765 28979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:23:07.641978 28979 hybrid_clock.cc:648] HybridClock initialized: now 1786083787641944 us; error 47 us; skew 500 ppm
Aug 07 06:23:07 dist-test-slave-44d2 krb5kdc[28140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083787, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:23:07.645768 28979 init.cc:377] Logged in from keytab as oryx/127.15.254.1@KRBTEST.COM (short username oryx)
I20260807 06:23:07.647046 28979 webserver.cc:460] Webserver started at http://127.15.254.1:41105/ using document root <none> and password file <none>
I20260807 06:23:07.647868 28979 fs_manager.cc:359] Metadata directory not provided
I20260807 06:23:07.647994 28979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:23:07.653641 28979 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:23:07.656543 29029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:07.658288 28979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 06:23:07.658414 28979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "10d001851f4c48cba68418e6537fbd98"
format_stamp: "Formatted at 2026-08-07 06:23:06 on dist-test-slave-44d2"
server_key: "d08d0c5d858793a8d135ed702fac7fba"
server_key_iv: "fb857571ed9fb1ca89067accbf708781"
server_key_version: "encryptionkey@0"
I20260807 06:23:07.659154 28979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:23:07.675194 28979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:23:07.678949 28979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:23:07.679180 28979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:23:07.679958 28979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:23:07.681030 28979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:23:07.681095 28979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:07.681147 28979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:23:07.681176 28979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:07.692858 28979 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:46683
I20260807 06:23:07.692922 29190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:46683 every 8 connection(s)
I20260807 06:23:07.694177 28979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:23:07.700397 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 28979
I20260807 06:23:07.700894 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:42111
--local_ip_for_outbound_sockets=127.15.254.2
--tserver_master_addrs=127.15.254.62:34449
--webserver_port=38985
--webserver_interface=127.15.254.2
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.254.2
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 06:23:07 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:07 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:07.709707 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:07.696832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.1:38903)
0807 06:23:07.697549 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:07.697602 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:07.698830 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:07.700236 (+ 1406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:07.700242 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:07.700737 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:07.701360 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:07.701379 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.703283 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.703287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:07.704169 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:07.704180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.705071 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.705076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:07.705279 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:07.705928 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:07.705946 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:07.708713 (+ 2767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":148,"threads_started":1}
W20260807 06:23:07.709697 29194 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:07.696711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:38903 (local address 127.15.254.62:34449)
0807 06:23:07.697245 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:07.697267 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:07.697757 (+ 490us) server_negotiation.cc:407] Connection header received
0807 06:23:07.699297 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:07.699314 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:07.699744 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:07.700157 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:07.701571 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.703117 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.704311 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.704879 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.705071 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:07.708934 (+ 3863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.1:38903: BlockingRecv error: recv got EOF from 127.15.254.1:38903 (error 108)
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":138,"threads_started":1}
W20260807 06:23:07.710537 29192 heartbeater.cc:644] Failed to heartbeat to 127.15.254.62:34449 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.254.62:34449: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Aug 07 06:23:07 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:07 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:07.717013 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:07.711074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.1:47509)
0807 06:23:07.711280 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:07.711297 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:07.711413 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:07.711697 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:07.711703 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:07.711898 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:07.712046 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:07.712056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.712856 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.712863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:07.713893 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:07.713908 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.714321 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.714325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:07.714435 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:07.714916 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:07.714932 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:07.716864 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123}
W20260807 06:23:07.717178 29194 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:07.711166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:47509 (local address 127.15.254.62:34449)
0807 06:23:07.711279 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:07.711282 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:07.711362 (+ 80us) server_negotiation.cc:407] Connection header received
0807 06:23:07.711523 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:07.711525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:07.711574 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:07.711641 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:07.712176 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.712742 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.714010 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.714217 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.714291 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:07.717009 (+ 2718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.1:47509: BlockingRecv error: recv got EOF from 127.15.254.1:47509 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 07 06:23:07 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:07 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:07.723433 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:07.717748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.1:51109)
0807 06:23:07.718258 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:07.718272 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:07.718359 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:07.718646 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:07.718649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:07.718812 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:07.718897 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:07.718903 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.719608 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.719612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:07.720534 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:07.720550 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.720974 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.720978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:07.721065 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:07.721504 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:07.721521 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:07.723333 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":366}
W20260807 06:23:07.723719 29194 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:07.717829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:51109 (local address 127.15.254.62:34449)
0807 06:23:07.717924 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:07.717927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:07.718346 (+ 419us) server_negotiation.cc:407] Connection header received
0807 06:23:07.718491 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:07.718495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:07.718543 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:07.718604 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:07.719009 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.719493 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.720661 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.720859 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.720937 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:07.723499 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.1:51109: BlockingRecv error: recv got EOF from 127.15.254.1:51109 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260807 06:23:07.723728 29192 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 06:23:07.823540 29196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:23:07.823815 29196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:23:07.823859 29196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:23:07.828173 29196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:23:07.828238 29196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:23:07.828343 29196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:23:07.834203 29196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.2.keytab
--principal=oryx/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:42111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=38985
--tserver_master_addrs=127.15.254.62:34449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:23:07.835259 29196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:23:07.836337 29196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:23:07.846755 29204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:07.846689 29206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:07.846801 29203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:07.847214 29196 server_base.cc:1034] running on GCE node
I20260807 06:23:07.847675 29196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:23:07.848305 29196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:23:07.849474 29196 hybrid_clock.cc:648] HybridClock initialized: now 1786083787849462 us; error 31 us; skew 500 ppm
Aug 07 06:23:07 dist-test-slave-44d2 krb5kdc[28140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083787, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:23:07.852437 29196 init.cc:377] Logged in from keytab as oryx/127.15.254.2@KRBTEST.COM (short username oryx)
I20260807 06:23:07.853511 29196 webserver.cc:460] Webserver started at http://127.15.254.2:38985/ using document root <none> and password file <none>
I20260807 06:23:07.854133 29196 fs_manager.cc:359] Metadata directory not provided
I20260807 06:23:07.854239 29196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:23:07.858239 29196 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260807 06:23:07.860337 29218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:07.861619 29196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 06:23:07.861761 29196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "ffdc6a236f24415bb138f4890b6d12f0"
format_stamp: "Formatted at 2026-08-07 06:23:06 on dist-test-slave-44d2"
server_key: "3a3e7782085afe4682a3581890c024a9"
server_key_iv: "e3907664b45aaedcf45a58665b69e6a7"
server_key_version: "encryptionkey@0"
I20260807 06:23:07.862310 29196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:23:07.890391 29196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:23:07.893755 29196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:23:07.893989 29196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:23:07.894636 29196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:23:07.895669 29196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:23:07.895746 29196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:07.895828 29196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:23:07.895872 29196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:07.906026 29196 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:42111
I20260807 06:23:07.906065 29390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:42111 every 8 connection(s)
I20260807 06:23:07.907034 29196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:23:07.909264 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 29196
I20260807 06:23:07.909799 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:43809
--local_ip_for_outbound_sockets=127.15.254.3
--tserver_master_addrs=127.15.254.62:34449
--webserver_port=34987
--webserver_interface=127.15.254.3
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.254.3
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 06:23:07 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:07 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
W20260807 06:23:07.919606 29194 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:07.908893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:46787 (local address 127.15.254.62:34449)
0807 06:23:07.909031 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:07.909035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:07.910327 (+ 1292us) server_negotiation.cc:407] Connection header received
0807 06:23:07.911744 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:07.911748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:07.911819 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:07.911928 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:07.913236 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.913802 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.915082 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.915315 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.915433 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:07.919474 (+ 4041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.2:46787: BlockingRecv error: recv got EOF from 127.15.254.2:46787 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 06:23:07.920269 29394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:07.909393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.2:46787)
0807 06:23:07.910156 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:07.910214 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:07.911508 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:07.912127 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:07.912134 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:07.912559 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:07.913079 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:07.913093 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.913940 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.913944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:07.914938 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:07.914964 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.915478 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.915482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:07.915663 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:07.916327 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:07.916344 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:07.919349 (+ 3005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":170,"threads_started":1}
W20260807 06:23:07.920763 29392 heartbeater.cc:644] Failed to heartbeat to 127.15.254.62:34449 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.254.62:34449: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Aug 07 06:23:07 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:07 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:07.927173 29394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:07.921265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.2:49225)
0807 06:23:07.921480 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:07.921497 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:07.921603 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:07.921874 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:07.921880 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:07.922081 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:07.922244 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:07.922253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.922982 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.922988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:07.923905 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:07.923916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.924334 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.924338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:07.924443 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:07.924968 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:07.924983 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:07.927007 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119}
W20260807 06:23:07.927268 29194 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:07.921358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:49225 (local address 127.15.254.62:34449)
0807 06:23:07.921478 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:07.921481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:07.921547 (+ 66us) server_negotiation.cc:407] Connection header received
0807 06:23:07.921706 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:07.921708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:07.921746 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:07.921813 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:07.922372 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.922869 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.924023 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.924228 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.924302 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:07.927144 (+ 2842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.2:49225: BlockingRecv error: recv got EOF from 127.15.254.2:49225 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 07 06:23:07 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:07 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:07.933593 29394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:07.927974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.2:40761)
0807 06:23:07.928072 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:07.928088 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:07.928167 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:07.928408 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:07.928410 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:07.928607 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:07.928683 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:07.928689 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.929762 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.929764 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:07.930268 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:07.930275 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.930882 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.930885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:07.930947 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:07.931472 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:07.931484 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:07.933481 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260807 06:23:07.933773 29194 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:07.928073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:40761 (local address 127.15.254.62:34449)
0807 06:23:07.928197 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:07.928201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:07.928214 (+ 13us) server_negotiation.cc:407] Connection header received
0807 06:23:07.928253 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:07.928256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:07.928297 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:07.928385 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:07.928810 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.929653 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.930416 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:07.930768 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:07.930873 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:07.933642 (+ 2769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.2:40761: BlockingRecv error: recv got EOF from 127.15.254.2:40761 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260807 06:23:07.933892 29392 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 06:23:08.033470 29395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:23:08.033751 29395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:23:08.033818 29395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:23:08.038110 29395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:23:08.038195 29395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:23:08.038295 29395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:23:08.043545 29395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/krb5kdc/oryx_127.15.254.3.keytab
--principal=oryx/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:43809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=34987
--tserver_master_addrs=127.15.254.62:34449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:23:08.044679 29395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:23:08.045711 29395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:23:08.055740 29402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:08.055745 29409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:08.055815 29395 server_base.cc:1034] running on GCE node
W20260807 06:23:08.055745 29403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:08.056380 29395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:23:08.057160 29395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:23:08.058329 29395 hybrid_clock.cc:648] HybridClock initialized: now 1786083788058315 us; error 24 us; skew 500 ppm
Aug 07 06:23:08 dist-test-slave-44d2 krb5kdc[28140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083788, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:23:08.061172 29395 init.cc:377] Logged in from keytab as oryx/127.15.254.3@KRBTEST.COM (short username oryx)
I20260807 06:23:08.062050 29395 webserver.cc:460] Webserver started at http://127.15.254.3:34987/ using document root <none> and password file <none>
I20260807 06:23:08.062641 29395 fs_manager.cc:359] Metadata directory not provided
I20260807 06:23:08.062762 29395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:23:08.066943 29395 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:23:08.069214 29418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:08.070389 29395 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 06:23:08.070531 29395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "6c0855cf31d543cb93578c3dcde059f4"
format_stamp: "Formatted at 2026-08-07 06:23:07 on dist-test-slave-44d2"
server_key: "1229dd808e68f0df26f4f8c6638e2ddc"
server_key_iv: "29b9dcc53fc87575aaf9ae9e55fcbfe6"
server_key_version: "encryptionkey@0"
I20260807 06:23:08.071099 29395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:23:08.096688 29395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:23:08.099897 29395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:23:08.100139 29395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:23:08.100858 29395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:23:08.101742 29395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:23:08.101820 29395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:08.101899 29395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:23:08.101944 29395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:08.112056 29395 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:43809
I20260807 06:23:08.112098 29569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:43809 every 8 connection(s)
I20260807 06:23:08.113157 29395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:23:08.117676 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 29395
Aug 07 06:23:08 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:08 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
W20260807 06:23:08.126188 29194 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:08.114979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:54293 (local address 127.15.254.62:34449)
0807 06:23:08.115123 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:08.115127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:08.115940 (+ 813us) server_negotiation.cc:407] Connection header received
0807 06:23:08.117490 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:08.117494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:08.117573 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:08.117681 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:08.119359 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:08.120231 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:08.121734 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:08.122080 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:08.122199 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:08.126014 (+ 3815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.3:54293: BlockingRecv error: recv got EOF from 127.15.254.3:54293 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 06:23:08.126920 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:08.115303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.3:54293)
0807 06:23:08.115778 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:08.115823 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:08.117248 (+ 1425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:08.117880 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:08.117887 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:08.118501 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:08.119154 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:08.119174 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:08.120371 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:08.120374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:08.121598 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:08.121608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:08.122233 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:08.122236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:08.122447 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:08.123223 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:08.123248 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:08.125907 (+ 2659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":112,"threads_started":1}
W20260807 06:23:08.127527 29572 heartbeater.cc:644] Failed to heartbeat to 127.15.254.62:34449 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.254.62:34449: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
I20260807 06:23:08.129415 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:08.118815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46328 (local address 127.15.254.62:34449)
0807 06:23:08.119103 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:08.119107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:08.119122 (+ 15us) server_negotiation.cc:407] Connection header received
0807 06:23:08.119159 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:08.119162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:08.119206 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:08.119297 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:08.120358 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:08.121150 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:08.122337 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:08.122690 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:08.122797 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:08.124194 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:08.124200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:08.124215 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:23:08.124253 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:08.126708 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:08.127278 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:08.127284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:08.127291 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:08.127437 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:08.127805 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:08.127809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:08.127810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:08.128314 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:08.128541 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:08.128986 (+ 445us) server_negotiation.cc:299] Negotiation successful
0807 06:23:08.129248 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":106,"threads_started":1}
Aug 07 06:23:08 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:08 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:08.134261 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:08.128023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.3:42461)
0807 06:23:08.128233 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:08.128257 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:08.128337 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:08.128582 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:08.128585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:08.128770 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:08.129002 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:08.129018 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:08.129962 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:08.129966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:08.130768 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:08.130779 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:08.131365 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:08.131369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:08.131477 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:08.132086 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:08.132101 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:08.134167 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132}
W20260807 06:23:08.134496 29194 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:08.128149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:42461 (local address 127.15.254.62:34449)
0807 06:23:08.128279 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:08.128283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:08.128303 (+ 20us) server_negotiation.cc:407] Connection header received
0807 06:23:08.128406 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:08.128409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:08.128451 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:08.128551 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:08.129141 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:08.129779 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:08.131000 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:08.131238 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:08.131372 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:08.134368 (+ 2996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.3:42461: BlockingRecv error: recv got EOF from 127.15.254.3:42461 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 07 06:23:08 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:08 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:08.140005 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:08.135007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.3:58385)
0807 06:23:08.135227 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:08.135239 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:08.135309 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:08.135514 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:08.135516 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:08.135640 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:08.135712 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:08.135717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:08.136453 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:08.136455 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:08.137003 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:08.137011 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:08.137453 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:08.137456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:08.137515 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:08.138054 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:08.138066 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:08.139875 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145}
W20260807 06:23:08.140190 29194 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:08.135117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:58385 (local address 127.15.254.62:34449)
0807 06:23:08.135247 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:08.135250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:08.135277 (+ 27us) server_negotiation.cc:407] Connection header received
0807 06:23:08.135386 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:08.135388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:08.135425 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:08.135488 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:08.135833 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:08.136337 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:08.137105 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:08.137303 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:08.137460 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:08.140049 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.3:58385: BlockingRecv error: recv got EOF from 127.15.254.3:58385 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260807 06:23:08.140268 29572 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 07 06:23:08 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:08 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:08.730331 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:08.724443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.1:48347)
0807 06:23:08.724795 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:08.724809 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:08.724922 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:08.725205 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:08.725207 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:08.725404 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:08.725510 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:08.725514 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:08.726483 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:08.726486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:08.727070 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:08.727080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:08.727582 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:08.727584 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:08.727653 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:08.728281 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:08.728293 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:08.730198 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":101,"threads_started":1}
W20260807 06:23:08.730525 29580 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:08.724532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:48347 (local address 127.15.254.62:34449)
0807 06:23:08.724809 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:08.724814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:08.724970 (+ 156us) server_negotiation.cc:407] Connection header received
0807 06:23:08.725021 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:08.725024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:08.725075 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:08.725171 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:08.725650 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:08.726368 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:08.727199 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:08.727453 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:08.727553 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:08.730345 (+ 2792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.1:48347: BlockingRecv error: recv got EOF from 127.15.254.1:48347 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":76,"threads_started":1}
Aug 07 06:23:08 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:08 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:08.941058 29581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:08.934613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.2:57375)
0807 06:23:08.934900 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:08.934914 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:08.935034 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:08.935463 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:08.935466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:08.935711 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:08.935818 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:08.935824 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:08.936959 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:08.936963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:08.937620 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:08.937629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:08.938122 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:08.938126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:08.938217 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:08.938798 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:08.938810 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:08.940917 (+ 2107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
W20260807 06:23:08.941185 29580 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:08.934717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:57375 (local address 127.15.254.62:34449)
0807 06:23:08.934965 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:08.934970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:08.934986 (+ 16us) server_negotiation.cc:407] Connection header received
0807 06:23:08.935265 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:08.935268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:08.935321 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:08.935399 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:08.936007 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:08.936833 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:08.937748 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:08.937996 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:08.938114 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:08.941061 (+ 2947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.2:57375: BlockingRecv error: recv got EOF from 127.15.254.2:57375 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 07 06:23:09 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:09 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:09.147327 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:09.140887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.3:38983)
0807 06:23:09.141169 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:09.141192 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:09.141289 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:09.141613 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:09.141615 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:09.141807 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:09.141910 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:09.141916 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:09.143221 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:09.143225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:09.143829 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:09.143840 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:09.144373 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:09.144376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:09.144443 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:09.145159 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:09.145171 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:09.147196 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":99,"threads_started":1}
W20260807 06:23:09.147573 29580 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:09.141055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:38983 (local address 127.15.254.62:34449)
0807 06:23:09.141209 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:09.141213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:09.141243 (+ 30us) server_negotiation.cc:407] Connection header received
0807 06:23:09.141427 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:09.141430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:09.141487 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:09.141572 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:09.142042 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:09.143057 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:09.144028 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:09.144254 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:09.144348 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:09.147405 (+ 3057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.3:38983: BlockingRecv error: recv got EOF from 127.15.254.3:38983 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 07 06:23:09 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:09 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:09.737100 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:09.731224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.1:35387)
0807 06:23:09.731566 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:09.731585 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:09.731709 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:09.731950 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:09.731952 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:09.732150 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:09.732254 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:09.732259 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:09.733099 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:09.733101 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:09.733658 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:09.733665 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:09.734137 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:09.734139 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:09.734201 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:09.734702 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:09.734714 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:09.736872 (+ 2158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":101,"threads_started":1}
W20260807 06:23:09.737139 29584 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:09.731438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:35387 (local address 127.15.254.62:34449)
0807 06:23:09.731705 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:09.731709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:09.731723 (+ 14us) server_negotiation.cc:407] Connection header received
0807 06:23:09.731782 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:09.731785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:09.731829 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:09.731905 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:09.732365 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:09.732971 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:09.733768 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:09.734017 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:09.734099 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:09.737013 (+ 2914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.1:35387: BlockingRecv error: recv got EOF from 127.15.254.1:35387 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":123,"threads_started":1}
Aug 07 06:23:09 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:09 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:09.948146 29585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:09.942075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.2:33943)
0807 06:23:09.942399 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:09.942413 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:09.942522 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:09.942796 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:09.942798 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:09.942992 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:09.943118 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:09.943124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:09.943986 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:09.943988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:09.944623 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:09.944632 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:09.945239 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:09.945242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:09.945312 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:09.945877 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:09.945890 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:09.947975 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":129,"threads_started":1}
W20260807 06:23:09.948355 29584 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:09.942121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:33943 (local address 127.15.254.62:34449)
0807 06:23:09.942316 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:09.942323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:09.942498 (+ 175us) server_negotiation.cc:407] Connection header received
0807 06:23:09.942615 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:09.942617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:09.942675 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:09.942776 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:09.943256 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:09.943813 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:09.944763 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:09.945109 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:09.945218 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:09.948160 (+ 2942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.2:33943: BlockingRecv error: recv got EOF from 127.15.254.2:33943 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 07 06:23:10 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:10 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:10.154194 29587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:10.148159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.3:45621)
0807 06:23:10.148461 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:10.148474 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:10.148616 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:10.148848 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:10.148856 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:10.149125 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:10.149279 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:10.149289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:10.150398 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:10.150401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:10.150966 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:10.150975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:10.151456 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:10.151466 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:10.151528 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:10.152029 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:10.152042 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:10.154050 (+ 2008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":116,"threads_started":1}
W20260807 06:23:10.154343 29584 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:10.148338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:45621 (local address 127.15.254.62:34449)
0807 06:23:10.148521 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:10.148528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:10.148568 (+ 40us) server_negotiation.cc:407] Connection header received
0807 06:23:10.148686 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:10.148688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:10.148736 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:10.148846 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:10.149401 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:10.150272 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:10.151116 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:10.151343 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:10.151437 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:10.154190 (+ 2753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.3:45621: BlockingRecv error: recv got EOF from 127.15.254.3:45621 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 07 06:23:10 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:10 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:10.744555 29590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:10.738065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.1:41497)
0807 06:23:10.738358 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:10.738379 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:10.738484 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:10.738944 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:10.738946 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:10.739152 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:10.739259 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:10.739265 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:10.740162 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:10.740165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:10.740826 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:10.740838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:10.741350 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:10.741373 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:10.741458 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:10.742161 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:10.742173 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:10.744351 (+ 2178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":100,"threads_started":1}
W20260807 06:23:10.744627 29592 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:10.738130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:41497 (local address 127.15.254.62:34449)
0807 06:23:10.738455 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:10.738459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:10.738476 (+ 17us) server_negotiation.cc:407] Connection header received
0807 06:23:10.738638 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:10.738665 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:10.738736 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:10.738869 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:10.739414 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:10.740011 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:10.740974 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:10.741231 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:10.741334 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:10.744477 (+ 3143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.1:41497: BlockingRecv error: recv got EOF from 127.15.254.1:41497 (error 108)
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":137,"threads_started":1}
Aug 07 06:23:10 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:10 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:10.955368 29593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:10.949230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.2:49327)
0807 06:23:10.949621 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:10.949636 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:10.949791 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:10.950143 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:10.950146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:10.950396 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:10.950498 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:10.950504 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:10.951317 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:10.951323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:10.951906 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:10.951914 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:10.952385 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:10.952388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:10.952522 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:10.953127 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:10.953140 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:10.955173 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":99,"threads_started":1}
W20260807 06:23:10.955439 29592 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:10.949382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:49327 (local address 127.15.254.62:34449)
0807 06:23:10.949584 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:10.949588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:10.949733 (+ 145us) server_negotiation.cc:407] Connection header received
0807 06:23:10.949962 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:10.949966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:10.950023 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:10.950118 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:10.950629 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:10.951200 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:10.952034 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:10.952258 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:10.952372 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:10.955312 (+ 2940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.2:49327: BlockingRecv error: recv got EOF from 127.15.254.2:49327 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 07 06:23:11 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:11 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:11.161015 29594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:11.155079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.3:58457)
0807 06:23:11.155406 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:11.155419 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:11.155552 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:11.155847 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:11.155850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:11.156081 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:11.156182 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:11.156188 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:11.157050 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:11.157055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:11.157636 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:11.157645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:11.158113 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:11.158116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:11.158206 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:11.158743 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:11.158755 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:11.160830 (+ 2075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":129,"threads_started":1}
W20260807 06:23:11.161067 29592 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:11.155134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:58457 (local address 127.15.254.62:34449)
0807 06:23:11.155301 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:11.155306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:11.155497 (+ 191us) server_negotiation.cc:407] Connection header received
0807 06:23:11.155665 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:11.155669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:11.155727 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:11.155816 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:11.156311 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:11.156930 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:11.157763 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:11.157990 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:11.158106 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:11.160961 (+ 2855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.3:58457: BlockingRecv error: recv got EOF from 127.15.254.3:58457 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 07 06:23:11 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:11 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:11.751024 29595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:11.745456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.1:35413)
0807 06:23:11.745730 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:11.745744 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:11.745837 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:11.746082 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:11.746084 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:11.746275 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:11.746378 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:11.746383 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:11.747204 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:11.747207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:11.747782 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:11.747791 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:11.748224 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:11.748227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:11.748295 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:11.748929 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:11.748943 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:11.750896 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":96,"threads_started":1}
W20260807 06:23:11.751250 29596 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:11.745520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:35413 (local address 127.15.254.62:34449)
0807 06:23:11.745777 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:11.745781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:11.745794 (+ 13us) server_negotiation.cc:407] Connection header received
0807 06:23:11.745917 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:11.745919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:11.745964 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:11.746032 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:11.746515 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:11.747072 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:11.747891 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:11.748113 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:11.748191 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:11.751074 (+ 2883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.1:35413: BlockingRecv error: recv got EOF from 127.15.254.1:35413 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":91,"threads_started":1}
Aug 07 06:23:11 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:11 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:11.962630 29597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:11.956543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.2:35269)
0807 06:23:11.956858 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:11.956875 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:11.956995 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:11.957308 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:11.957311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:11.957505 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:11.957610 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:11.957616 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:11.958485 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:11.958489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:11.959077 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:11.959086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:11.959591 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:11.959594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:11.959660 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:11.960249 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:11.960262 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:11.962436 (+ 2174us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":124,"threads_started":1}
W20260807 06:23:11.962698 29596 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:11.956605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:35269 (local address 127.15.254.62:34449)
0807 06:23:11.956752 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:11.956756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:11.956970 (+ 214us) server_negotiation.cc:407] Connection header received
0807 06:23:11.957085 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:11.957088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:11.957137 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:11.957213 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:11.957731 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:11.958339 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:11.959220 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:11.959474 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:11.959582 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:11.962584 (+ 3002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.2:35269: BlockingRecv error: recv got EOF from 127.15.254.2:35269 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 07 06:23:12 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:12 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:12.168843 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:12.161944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.3:54167)
0807 06:23:12.162359 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:12.162373 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:12.162564 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:12.162879 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:12.162882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:12.163186 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:12.163300 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:12.163306 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:12.164167 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:12.164173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:12.164996 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:12.165007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:12.165540 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:12.165544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:12.165641 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:12.166349 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:12.166363 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:12.168723 (+ 2360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":126,"threads_started":1}
W20260807 06:23:12.169106 29596 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:12.162026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:54167 (local address 127.15.254.62:34449)
0807 06:23:12.162207 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:12.162215 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:12.162487 (+ 272us) server_negotiation.cc:407] Connection header received
0807 06:23:12.162689 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:12.162695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:12.162752 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:12.162849 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:12.163431 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:12.164029 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:12.165146 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:12.165391 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:12.165536 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:12.168956 (+ 3420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.3:54167: BlockingRecv error: recv got EOF from 127.15.254.3:54167 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 07 06:23:12 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:12 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:12.757893 29599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:12.752002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.1:45033)
0807 06:23:12.752309 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:12.752326 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:12.752436 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:12.752843 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:12.752846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:12.753106 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:12.753249 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:12.753257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:12.754212 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:12.754215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:12.754779 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:12.754787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:12.755267 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:12.755269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:12.755334 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:12.755849 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:12.755861 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:12.757776 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":117,"threads_started":1}
W20260807 06:23:12.758143 29600 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:12.752140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:45033 (local address 127.15.254.62:34449)
0807 06:23:12.752439 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:12.752445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:12.752460 (+ 15us) server_negotiation.cc:407] Connection header received
0807 06:23:12.752610 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:12.752614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:12.752678 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:12.752791 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:12.753379 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:12.754106 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:12.754938 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:12.755161 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:12.755246 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:12.758005 (+ 2759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.1:45033: BlockingRecv error: recv got EOF from 127.15.254.1:45033 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":79,"threads_started":1}
Aug 07 06:23:12 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:12 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:12.971976 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:12.963756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.2:54285)
0807 06:23:12.965946 (+ 2190us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:12.965963 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:12.966138 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:12.966477 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:12.966480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:12.966766 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:12.966900 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:12.966908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:12.967817 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:12.967834 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:12.968476 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:12.968552 (+ 76us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:12.969126 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:12.969131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:12.969235 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:12.969785 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:12.969802 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:12.971847 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2099,"thread_start_us":1958,"threads_started":1}
W20260807 06:23:12.972254 29600 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:12.963951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:54285 (local address 127.15.254.62:34449)
0807 06:23:12.964177 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:12.964186 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:12.966074 (+ 1888us) server_negotiation.cc:407] Connection header received
0807 06:23:12.966261 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:12.966266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:12.966341 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:12.966451 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:12.967058 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:12.967679 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:12.968694 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:12.968968 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:12.969119 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:12.972122 (+ 3003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.2:54285: BlockingRecv error: recv got EOF from 127.15.254.2:54285 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 07 06:23:13 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:13 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:13.175755 29603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:13.169708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.3:35033)
0807 06:23:13.170089 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:13.170104 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:13.170216 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:13.170504 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:13.170506 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:13.170716 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:13.170825 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:13.170830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:13.171787 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:13.171790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:13.172366 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:13.172377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:13.172959 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:13.172962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:13.173026 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:13.173642 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:13.173654 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:13.175617 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":129,"threads_started":1}
W20260807 06:23:13.175971 29600 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:13.169798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:35033 (local address 127.15.254.62:34449)
0807 06:23:13.170073 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:13.170077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:13.170190 (+ 113us) server_negotiation.cc:407] Connection header received
0807 06:23:13.170305 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:13.170307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:13.170361 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:13.170443 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:13.171089 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:13.171656 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:13.172594 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:13.172835 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:13.172939 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:13.175782 (+ 2843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.3:35033: BlockingRecv error: recv got EOF from 127.15.254.3:35033 (error 108)
Metrics: {"server-negotiator.queue_time_us":159}
Aug 07 06:23:13 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:13 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:13.764358 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:13.758726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.1:49779)
0807 06:23:13.759020 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:13.759036 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:13.759148 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:13.759384 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:13.759386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:13.759579 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:13.759680 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:13.759685 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:13.760441 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:13.760444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:13.761021 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:13.761029 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:13.761455 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:13.761458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:13.761519 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:13.762005 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:13.762018 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:13.764227 (+ 2209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":105,"threads_started":1}
W20260807 06:23:13.764640 29607 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:13.758807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:49779 (local address 127.15.254.62:34449)
0807 06:23:13.759107 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:13.759111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:13.759125 (+ 14us) server_negotiation.cc:407] Connection header received
0807 06:23:13.759239 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:13.759242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:13.759288 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:13.759373 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:13.759783 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:13.760338 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:13.761128 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:13.761353 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:13.761432 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:13.764404 (+ 2972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.1:49779: BlockingRecv error: recv got EOF from 127.15.254.1:49779 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":141,"threads_started":1}
Aug 07 06:23:13 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:13 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:13.979758 29608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:13.972907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.2:52257)
0807 06:23:13.973216 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:13.973230 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:13.973343 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:13.973616 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:13.973618 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:13.973837 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:13.973958 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:13.973965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:13.975066 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:13.975072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:13.975932 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:13.975943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:13.976436 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:13.976440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:13.976551 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:13.977073 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:13.977086 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:13.979637 (+ 2551us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":102,"threads_started":1}
W20260807 06:23:13.979934 29607 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:13.972999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:52257 (local address 127.15.254.62:34449)
0807 06:23:13.973195 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:13.973199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:13.973306 (+ 107us) server_negotiation.cc:407] Connection header received
0807 06:23:13.973448 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:13.973451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:13.973498 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:13.973571 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:13.974107 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:13.974924 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:13.976068 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:13.976313 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:13.976428 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:13.979807 (+ 3379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.2:52257: BlockingRecv error: recv got EOF from 127.15.254.2:52257 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 07 06:23:14 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:14 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:14.183171 29609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:14.176723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.3:33895)
0807 06:23:14.177205 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:14.177224 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:14.177344 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:14.177641 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:14.177644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:14.177872 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:14.177974 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:14.177979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:14.178770 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:14.178775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:14.179347 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:14.179357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:14.179870 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:14.179873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:14.179962 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:14.180608 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:14.180620 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:14.182972 (+ 2352us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":168,"threads_started":1}
W20260807 06:23:14.183274 29607 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:14.177183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:33895 (local address 127.15.254.62:34449)
0807 06:23:14.177315 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:14.177319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:14.177334 (+ 15us) server_negotiation.cc:407] Connection header received
0807 06:23:14.177439 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:14.177444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:14.177493 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:14.177599 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:14.178107 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:14.178653 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:14.179497 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:14.179749 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:14.179864 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:14.183123 (+ 3259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.3:33895: BlockingRecv error: recv got EOF from 127.15.254.3:33895 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 07 06:23:14 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:14 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:14.771462 29610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:14.765188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.1:40461)
0807 06:23:14.765600 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:14.765615 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:14.765724 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:14.766039 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:14.766041 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:14.766298 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:14.766398 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:14.766405 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:14.767407 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:14.767411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:14.768154 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:14.768165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:14.768640 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:14.768643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:14.768710 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:14.769264 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:14.769277 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:14.771293 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":139,"threads_started":1}
W20260807 06:23:14.771643 29611 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:14.765293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:40461 (local address 127.15.254.62:34449)
0807 06:23:14.765600 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:14.765604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:14.765715 (+ 111us) server_negotiation.cc:407] Connection header received
0807 06:23:14.765843 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:14.765850 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:14.765921 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:14.765997 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:14.766588 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:14.767277 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:14.768268 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:14.768510 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:14.768627 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:14.771487 (+ 2860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.1:40461: BlockingRecv error: recv got EOF from 127.15.254.1:40461 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":94,"threads_started":1}
Aug 07 06:23:14 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:14 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:14.993373 29612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:14.981020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.2:40431)
0807 06:23:14.981380 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:14.981394 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:14.981500 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:14.981838 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:14.981841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:14.982115 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:14.982249 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:14.982261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:14.983479 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:14.983484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:14.984400 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:14.984413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:14.985241 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:14.985245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:14.985320 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:14.991072 (+ 5752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:14.991087 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:14.993242 (+ 2155us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":95,"threads_started":1}
W20260807 06:23:14.993587 29611 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:14.981040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:40431 (local address 127.15.254.62:34449)
0807 06:23:14.981226 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:14.981232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:14.981478 (+ 246us) server_negotiation.cc:407] Connection header received
0807 06:23:14.981623 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:14.981628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:14.981686 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:14.981794 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:14.982405 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:14.983326 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:14.984564 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:14.984915 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:14.985017 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:14.993425 (+ 8408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.2:40431: BlockingRecv error: recv got EOF from 127.15.254.2:40431 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 07 06:23:15 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:15 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:15.190250 29614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:15.184131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.3:57939)
0807 06:23:15.184499 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:15.184534 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:15.184672 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:15.184967 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:15.184970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:15.185199 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:15.185303 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:15.185309 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:15.186093 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:15.186099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:15.186668 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:15.186677 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:15.187151 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:15.187155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:15.187243 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:15.187764 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:15.187776 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:15.190068 (+ 2292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":98,"threads_started":1}
W20260807 06:23:15.190306 29611 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:15.184193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:57939 (local address 127.15.254.62:34449)
0807 06:23:15.184329 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:15.184334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:15.184623 (+ 289us) server_negotiation.cc:407] Connection header received
0807 06:23:15.184783 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:15.184787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:15.184844 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:15.184934 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:15.185433 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:15.185972 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:15.186791 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:15.187032 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:15.187145 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:15.190197 (+ 3052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.3:57939: BlockingRecv error: recv got EOF from 127.15.254.3:57939 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 07 06:23:15 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:15 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:15.778436 29615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:15.772614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.1:36965)
0807 06:23:15.773091 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:15.773106 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:15.773205 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:15.773503 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:15.773506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:15.773707 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:15.773835 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:15.773841 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:15.774672 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:15.774675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:15.775364 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:15.775375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:15.775834 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:15.775837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:15.775901 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:15.776449 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:15.776461 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:15.778317 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":132,"threads_started":1}
W20260807 06:23:15.778610 29616 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:15.772776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:36965 (local address 127.15.254.62:34449)
0807 06:23:15.773181 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:15.773185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:15.773199 (+ 14us) server_negotiation.cc:407] Connection header received
0807 06:23:15.773289 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:15.773291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:15.773335 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:15.773419 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:15.773988 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:15.774555 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:15.775476 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:15.775726 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:15.775809 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:15.778462 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.1:36965: BlockingRecv error: recv got EOF from 127.15.254.1:36965 (error 108)
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":86,"threads_started":1}
Aug 07 06:23:15 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:15 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:16.000723 29618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:15.994357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.2:56097)
0807 06:23:15.994670 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:15.994685 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:15.994802 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:15.995112 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:15.995115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:15.995304 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:15.995409 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:15.995415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:15.996788 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:15.996792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:15.997496 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:15.997507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:15.997951 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:15.997954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:15.998016 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:15.998519 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:15.998531 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:16.000569 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":129,"threads_started":1}
W20260807 06:23:16.000892 29616 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:15.994448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:56097 (local address 127.15.254.62:34449)
0807 06:23:15.994610 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:15.994614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:15.994769 (+ 155us) server_negotiation.cc:407] Connection header received
0807 06:23:15.994920 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:15.994922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:15.994974 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:15.995060 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:15.995546 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:15.996660 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:15.997611 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:15.997844 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:15.997930 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:16.000761 (+ 2831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.2:56097: BlockingRecv error: recv got EOF from 127.15.254.2:56097 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 07 06:23:16 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:16 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:16.196998 29621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:16.191110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.3:40443)
0807 06:23:16.191433 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:16.191447 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:16.191551 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:16.191822 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:16.191825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:16.192022 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:16.192160 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:16.192167 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:16.192982 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:16.192985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:16.193569 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:16.193578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:16.194049 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:16.194051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:16.194137 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:16.194644 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:16.194655 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:16.196870 (+ 2215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":139,"threads_started":1}
W20260807 06:23:16.197161 29616 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:16.191247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:40443 (local address 127.15.254.62:34449)
0807 06:23:16.191392 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:16.191395 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:16.191519 (+ 124us) server_negotiation.cc:407] Connection header received
0807 06:23:16.191634 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:16.191636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:16.191685 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:16.191770 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:16.192282 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:16.192879 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:16.193687 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:16.193911 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:16.194029 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:16.197048 (+ 3019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.3:40443: BlockingRecv error: recv got EOF from 127.15.254.3:40443 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 07 06:23:16 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:16 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:16.785346 29623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:16.779328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.1:40129)
0807 06:23:16.779638 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:16.779652 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:16.779752 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:16.780053 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:16.780055 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:16.780282 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:16.780386 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:16.780391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:16.781352 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:16.781355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:16.781917 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:16.781925 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:16.782401 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:16.782404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:16.782471 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:16.783033 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:16.783045 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:16.785142 (+ 2097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":110,"threads_started":1}
W20260807 06:23:16.785421 29627 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:16.779476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:40129 (local address 127.15.254.62:34449)
0807 06:23:16.779728 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:16.779732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:16.779745 (+ 13us) server_negotiation.cc:407] Connection header received
0807 06:23:16.779834 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:16.779836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:16.779880 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:16.780024 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:16.780562 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:16.781229 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:16.782048 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:16.782281 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:16.782375 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:16.785279 (+ 2904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.1:40129: BlockingRecv error: recv got EOF from 127.15.254.1:40129 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":93,"threads_started":1}
Aug 07 06:23:17 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:17 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:17.007893 29629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:17.001587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.2:57573)
0807 06:23:17.001944 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:17.001964 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:17.002089 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:17.002314 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:17.002315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:17.002511 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:17.002622 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:17.002628 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:17.003428 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:17.003431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:17.004014 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:17.004022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:17.004596 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:17.004601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:17.004681 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:17.005344 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:17.005357 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:17.007669 (+ 2312us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":88,"threads_started":1}
W20260807 06:23:17.007962 29627 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:17.001703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:57573 (local address 127.15.254.62:34449)
0807 06:23:17.001852 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:17.001858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:17.002050 (+ 192us) server_negotiation.cc:407] Connection header received
0807 06:23:17.002184 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:17.002186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:17.002233 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:17.002304 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:17.002731 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:17.003317 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:17.004134 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:17.004378 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:17.004472 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:17.007833 (+ 3361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.2:57573: BlockingRecv error: recv got EOF from 127.15.254.2:57573 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 07 06:23:17 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:17 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:17.204164 29631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:17.197975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.3:33165)
0807 06:23:17.198276 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:17.198290 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:17.198389 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:17.198625 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:17.198628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:17.198816 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:17.198970 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:17.198976 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:17.199813 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:17.199818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:17.200714 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:17.200726 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:17.201321 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:17.201325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:17.201417 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:17.201948 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:17.201991 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:17.203939 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":98,"threads_started":1}
W20260807 06:23:17.204176 29627 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:17.198156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:33165 (local address 127.15.254.62:34449)
0807 06:23:17.198333 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:17.198336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:17.198351 (+ 15us) server_negotiation.cc:407] Connection header received
0807 06:23:17.198462 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:17.198465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:17.198510 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:17.198579 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:17.199111 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:17.199671 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:17.200851 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:17.201191 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:17.201316 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:17.204072 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.3:33165: BlockingRecv error: recv got EOF from 127.15.254.3:33165 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 07 06:23:17 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:17 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:17.792230 29632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:17.786468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.1:57957)
0807 06:23:17.786786 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:17.786801 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:17.786956 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:17.787253 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:17.787255 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:17.787453 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:17.787557 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:17.787562 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:17.788337 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:17.788340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:17.788961 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:17.788970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:17.789452 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:17.789455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:17.789522 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:17.790137 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:17.790150 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:17.792113 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":103,"threads_started":1}
W20260807 06:23:17.792431 29634 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:17.786540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:57957 (local address 127.15.254.62:34449)
0807 06:23:17.786896 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:17.786900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:17.786916 (+ 16us) server_negotiation.cc:407] Connection header received
0807 06:23:17.787036 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:17.787038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:17.787106 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:17.787196 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:17.787669 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:17.788222 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:17.789101 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:17.789334 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:17.789419 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:17.792251 (+ 2832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.1:57957: BlockingRecv error: recv got EOF from 127.15.254.1:57957 (error 108)
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":165,"threads_started":1}
Aug 07 06:23:18 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:18 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:18.015136 29635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:18.008939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.2:39501)
0807 06:23:18.009351 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:18.009371 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:18.009517 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:18.009807 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:18.009811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:18.010043 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:18.010144 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:18.010150 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:18.010941 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:18.010945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:18.011553 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:18.011562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:18.012039 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:18.012043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:18.012138 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:18.012829 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:18.012842 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:18.014963 (+ 2121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":103,"threads_started":1}
W20260807 06:23:18.015385 29634 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:18.009020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:39501 (local address 127.15.254.62:34449)
0807 06:23:18.009305 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:18.009310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:18.009457 (+ 147us) server_negotiation.cc:407] Connection header received
0807 06:23:18.009628 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:18.009633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:18.009687 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:18.009782 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:18.010268 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:18.010807 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:18.011680 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:18.011916 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:18.012032 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:18.015224 (+ 3192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.2:39501: BlockingRecv error: recv got EOF from 127.15.254.2:39501 (error 108)
Metrics: {"server-negotiator.queue_time_us":172}
Aug 07 06:23:18 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:18 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:18.211619 29639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:18.205313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.3:52931)
0807 06:23:18.205700 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:18.205717 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:18.205845 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:18.206212 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:18.206215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:18.206493 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:18.206608 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:18.206615 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:18.207413 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:18.207416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:18.208076 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:18.208088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:18.208609 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:18.208613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:18.208692 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:18.209274 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:18.209287 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:18.211376 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":88,"threads_started":1}
W20260807 06:23:18.211727 29634 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:18.205477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:52931 (local address 127.15.254.62:34449)
0807 06:23:18.205702 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:18.205709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:18.205796 (+ 87us) server_negotiation.cc:407] Connection header received
0807 06:23:18.205957 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:18.205962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:18.206044 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:18.206128 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:18.206724 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:18.207270 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:18.208200 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:18.208424 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:18.208586 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:18.211572 (+ 2986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.3:52931: BlockingRecv error: recv got EOF from 127.15.254.3:52931 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Aug 07 06:23:18 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:18 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:18.798575 29640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:18.793122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.1:54175)
0807 06:23:18.793390 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:18.793403 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:18.793494 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:18.793721 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:18.793723 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:18.793907 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:18.794006 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:18.794011 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:18.794785 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:18.794787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:18.795348 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:18.795356 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:18.795808 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:18.795810 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:18.795878 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:18.796472 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:18.796543 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:18.798448 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":88,"threads_started":1}
W20260807 06:23:18.798796 29641 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:18.793170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:54175 (local address 127.15.254.62:34449)
0807 06:23:18.793445 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:18.793448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:18.793462 (+ 14us) server_negotiation.cc:407] Connection header received
0807 06:23:18.793584 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:18.793587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:18.793634 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:18.793709 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:18.794119 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:18.794686 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:18.795463 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:18.795704 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:18.795784 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:18.798625 (+ 2841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.1:54175: BlockingRecv error: recv got EOF from 127.15.254.1:54175 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":66,"threads_started":1}
Aug 07 06:23:19 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:19 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:19.022737 29642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:19.016241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.2:48453)
0807 06:23:19.016651 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:19.016674 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:19.016878 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:19.017191 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:19.017195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:19.017427 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:19.017530 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:19.017535 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:19.018613 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:19.018617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:19.019303 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:19.019312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:19.019840 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:19.019844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:19.019937 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:19.020559 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:19.020571 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:19.022555 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":166,"threads_started":1}
W20260807 06:23:19.022807 29641 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:19.016351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:48453 (local address 127.15.254.62:34449)
0807 06:23:19.016607 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:19.016613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:19.016801 (+ 188us) server_negotiation.cc:407] Connection header received
0807 06:23:19.016989 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:19.016992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:19.017063 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:19.017163 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:19.017681 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:19.018462 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:19.019427 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:19.019710 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:19.019819 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:19.022693 (+ 2874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.2:48453: BlockingRecv error: recv got EOF from 127.15.254.2:48453 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 07 06:23:19 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:19 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:19.219211 29643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:19.212774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.3:49949)
0807 06:23:19.213227 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:19.213249 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:19.213396 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:19.213695 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:19.213698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:19.213924 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:19.214025 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:19.214032 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:19.214836 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:19.214842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:19.215418 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:19.215427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:19.215915 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:19.215919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:19.216015 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:19.216661 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:19.216676 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:19.219017 (+ 2341us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":118,"threads_started":1}
W20260807 06:23:19.219385 29641 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:19.212989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:49949 (local address 127.15.254.62:34449)
0807 06:23:19.213195 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:19.213200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:19.213330 (+ 130us) server_negotiation.cc:407] Connection header received
0807 06:23:19.213504 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:19.213508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:19.213568 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:19.213661 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:19.214159 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:19.214710 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:19.215549 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:19.215783 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:19.215901 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:19.219235 (+ 3334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.3:49949: BlockingRecv error: recv got EOF from 127.15.254.3:49949 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 07 06:23:19 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:19 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:19.809085 29644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:19.799505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.1:52199)
0807 06:23:19.799856 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:19.799871 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:19.799984 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:19.800274 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:19.800276 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:19.800471 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:19.800602 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:19.800608 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:19.801449 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:19.801452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:19.802015 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:19.802022 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:19.802499 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:19.802503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:19.802602 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:19.806708 (+ 4106us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:19.806721 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:19.808949 (+ 2228us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":116,"threads_started":1}
W20260807 06:23:19.809298 29645 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:19.799695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:52199 (local address 127.15.254.62:34449)
0807 06:23:19.800049 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:19.800053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:19.800067 (+ 14us) server_negotiation.cc:407] Connection header received
0807 06:23:19.800104 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:19.800106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:19.800147 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:19.800227 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:19.800719 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:19.801335 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:19.802136 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:19.802367 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:19.802487 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:19.809141 (+ 6654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.1:52199: BlockingRecv error: recv got EOF from 127.15.254.1:52199 (error 108)
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":89,"threads_started":1}
Aug 07 06:23:20 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:20 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:20.033766 29647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:20.023606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.2:39797)
0807 06:23:20.023912 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:20.023930 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:20.024065 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:20.024313 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:20.024315 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:20.024618 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:20.024727 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:20.024734 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:20.025514 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:20.025518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:20.026133 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:20.026143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:20.026559 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:20.026562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:20.026628 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:20.031413 (+ 4785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:20.031427 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:20.033585 (+ 2158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":117,"threads_started":1}
W20260807 06:23:20.033816 29645 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:20.023746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:39797 (local address 127.15.254.62:34449)
0807 06:23:20.023897 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:20.023901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:20.024033 (+ 132us) server_negotiation.cc:407] Connection header received
0807 06:23:20.024151 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:20.024153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:20.024198 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:20.024273 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:20.024841 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:20.025401 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:20.026242 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:20.026455 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:20.026533 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:20.033713 (+ 7180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.2:39797: BlockingRecv error: recv got EOF from 127.15.254.2:39797 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 07 06:23:20 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:20 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:20.226083 29648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:20.220176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.3:48275)
0807 06:23:20.220520 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:20.220532 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:20.220668 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:20.221025 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:20.221029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:20.221243 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:20.221351 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:20.221356 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:20.222161 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:20.222166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:20.222746 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:20.222755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:20.223233 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:20.223237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:20.223329 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:20.223863 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:20.223926 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:20.225846 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":96,"threads_started":1}
W20260807 06:23:20.226106 29645 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:20.220231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:48275 (local address 127.15.254.62:34449)
0807 06:23:20.220414 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:20.220419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:20.220612 (+ 193us) server_negotiation.cc:407] Connection header received
0807 06:23:20.220779 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:20.220784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:20.220841 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:20.220914 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:20.221484 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:20.222043 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:20.222875 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:20.223109 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:20.223228 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:20.225995 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.3:48275: BlockingRecv error: recv got EOF from 127.15.254.3:48275 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 07 06:23:20 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:20 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:20.816171 29649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:20.809989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.1:43075)
0807 06:23:20.810286 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:20.810300 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:20.810389 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:20.810714 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:20.810717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:20.811020 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:20.811139 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:20.811146 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:20.811978 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:20.811981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:20.812713 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:20.812724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:20.813217 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:20.813221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:20.813296 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:20.813848 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:20.813860 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:20.815994 (+ 2134us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":108,"threads_started":1}
W20260807 06:23:20.816319 29651 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:20.810074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:43075 (local address 127.15.254.62:34449)
0807 06:23:20.810321 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:20.810324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:20.810357 (+ 33us) server_negotiation.cc:407] Connection header received
0807 06:23:20.810483 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:20.810486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:20.810533 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:20.810613 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:20.811266 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:20.811816 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:20.812839 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:20.813091 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:20.813175 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:20.816171 (+ 2996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.1:43075: BlockingRecv error: recv got EOF from 127.15.254.1:43075 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":91,"threads_started":1}
Aug 07 06:23:21 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:21 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:21.040637 29652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:21.034597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.2:43997)
0807 06:23:21.034926 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:21.034943 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:21.035108 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:21.035388 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:21.035391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:21.035606 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:21.035703 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:21.035708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:21.036522 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:21.036525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:21.037117 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:21.037126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:21.037633 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:21.037637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:21.037724 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:21.038287 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:21.038300 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:21.040433 (+ 2133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":123,"threads_started":1}
W20260807 06:23:21.040810 29651 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:21.034764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:43997 (local address 127.15.254.62:34449)
0807 06:23:21.034927 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:21.034934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:21.035021 (+ 87us) server_negotiation.cc:407] Connection header received
0807 06:23:21.035221 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:21.035224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:21.035275 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:21.035360 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:21.035824 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:21.036377 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:21.037245 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:21.037510 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:21.037631 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:21.040650 (+ 3019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.2:43997: BlockingRecv error: recv got EOF from 127.15.254.2:43997 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 07 06:23:21 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:21 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:21.232457 29653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:21.226920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.3:57389)
0807 06:23:21.227207 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:21.227220 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:21.227319 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:21.227561 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:21.227563 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:21.227761 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:21.227862 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:21.227867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:21.228704 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:21.228707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:21.229341 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:21.229352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:21.229800 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:21.229803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:21.229870 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:21.230424 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:21.230436 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:21.232331 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":109,"threads_started":1}
W20260807 06:23:21.232806 29651 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:21.227003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:57389 (local address 127.15.254.62:34449)
0807 06:23:21.227149 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:21.227152 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:21.227289 (+ 137us) server_negotiation.cc:407] Connection header received
0807 06:23:21.227397 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:21.227399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:21.227446 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:21.227516 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:21.228009 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:21.228599 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:21.229459 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:21.229691 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:21.229773 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:21.232672 (+ 2899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.3:57389: BlockingRecv error: recv got EOF from 127.15.254.3:57389 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 07 06:23:21 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:21 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:21.823184 29654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:21.817057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.1:58607)
0807 06:23:21.817475 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:21.817489 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:21.817606 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:21.817870 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:21.817872 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:21.818071 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:21.818182 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:21.818187 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:21.819108 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:21.819114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:21.819715 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:21.819724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:21.820222 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:21.820226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:21.820317 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:21.820938 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:21.820951 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:21.823048 (+ 2097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":117,"threads_started":1}
W20260807 06:23:21.823405 29655 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:21.817160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:58607 (local address 127.15.254.62:34449)
0807 06:23:21.817529 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:21.817534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:21.817557 (+ 23us) server_negotiation.cc:407] Connection header received
0807 06:23:21.817695 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:21.817698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:21.817744 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:21.817842 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:21.818328 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:21.818981 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:21.819849 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:21.820094 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:21.820214 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:21.823238 (+ 3024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.1:58607: BlockingRecv error: recv got EOF from 127.15.254.1:58607 (error 108)
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":74,"threads_started":1}
Aug 07 06:23:22 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:22 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:22.047755 29656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:22.041472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.2:43393)
0807 06:23:22.041820 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:22.041834 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:22.042006 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:22.042322 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:22.042325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:22.042525 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:22.042631 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:22.042637 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:22.043595 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:22.043599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:22.044192 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:22.044202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:22.044757 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:22.044761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:22.044851 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:22.045436 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:22.045449 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:22.047541 (+ 2092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":130,"threads_started":1}
W20260807 06:23:22.047782 29655 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:22.041572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:43393 (local address 127.15.254.62:34449)
0807 06:23:22.041740 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:22.041747 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:22.041965 (+ 218us) server_negotiation.cc:407] Connection header received
0807 06:23:22.042117 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:22.042120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:22.042171 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:22.042243 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:22.042768 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:22.043470 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:22.044329 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:22.044636 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:22.044751 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:22.047678 (+ 2927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.2:43393: BlockingRecv error: recv got EOF from 127.15.254.2:43393 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 07 06:23:22 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:22 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.3@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:22.239446 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:22.233477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.3:48917)
0807 06:23:22.233786 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:22.233800 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:22.233963 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:22.234205 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:22.234207 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:22.234409 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:22.234514 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:22.234520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:22.235282 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:22.235284 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:22.235962 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:22.235975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:22.236531 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:22.236538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:22.236621 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:22.237110 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:22.237123 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:22.239298 (+ 2175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":102,"threads_started":1}
W20260807 06:23:22.239580 29655 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:22.233527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:48917 (local address 127.15.254.62:34449)
0807 06:23:22.233668 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:22.233671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:22.233857 (+ 186us) server_negotiation.cc:407] Connection header received
0807 06:23:22.234038 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:22.234040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:22.234090 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:22.234159 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:22.234629 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:22.235169 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:22.236108 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:22.236363 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:22.236448 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:22.239448 (+ 3000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.3:48917: BlockingRecv error: recv got EOF from 127.15.254.3:48917 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 07 06:23:22 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:22 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.1@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:22.830341 29660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:22.824285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.1:44255)
0807 06:23:22.824654 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:22.824671 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:22.824800 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:22.825263 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:22.825266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:22.825484 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:22.825587 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:22.825593 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:22.826419 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:22.826422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:22.826982 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:22.826990 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:22.827427 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:22.827430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:22.827495 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:22.828100 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:22.828113 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:22.830142 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":86,"threads_started":1}
W20260807 06:23:22.830412 29663 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:22.824424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:44255 (local address 127.15.254.62:34449)
0807 06:23:22.824712 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:22.824719 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:22.824741 (+ 22us) server_negotiation.cc:407] Connection header received
0807 06:23:22.825029 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:22.825033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:22.825098 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:22.825210 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:22.825706 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:22.826308 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:22.827096 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:22.827322 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:22.827411 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:22.830266 (+ 2855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.1:44255: BlockingRecv error: recv got EOF from 127.15.254.1:44255 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":88,"threads_started":1}
Aug 07 06:23:23 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:23:23 dist-test-slave-44d2 krb5kdc[28140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.254.2@KRBTEST.COM for oryx/127.15.254.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:23:23.054891 29664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:23.048790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:34449 (local address 127.15.254.2:42419)
0807 06:23:23.049140 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:23.049154 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:23.049285 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:23.049675 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:23.049678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:23.049988 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:23.050121 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:23.050131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:23.051204 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:23.051207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:23.051759 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:23.051766 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:23.052187 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:23.052190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:23.052255 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:23.052798 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:23.052810 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:23.054759 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.254.62:34449: Server oryx/127.15.254.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":128,"threads_started":1}
W20260807 06:23:23.055153 29663 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:23.048885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:42419 (local address 127.15.254.62:34449)
0807 06:23:23.049071 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:23.049076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:23.049227 (+ 151us) server_negotiation.cc:407] Connection header received
0807 06:23:23.049391 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:23.049397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:23.049464 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:23.049582 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:23.050257 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:23.051095 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:23.051862 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:23.052084 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:23.052166 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:23.054900 (+ 2734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.254.2:42419: BlockingRecv error: recv got EOF from 127.15.254.2:42419 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 06:23:23.119684 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 28979
I20260807 06:23:23.132190 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 29196
I20260807 06:23:23.145139 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 29395
I20260807 06:23:23.163776 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 28878
2026-08-07T06:23:23Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18425 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-07T06:23:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:23:23Z Disabled control of system clock
I20260807 06:23:23.201277 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:44385
--webserver_interface=127.15.254.62
--webserver_port=0
--builtin_ntp_servers=127.15.254.20:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:44385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260807 06:23:23.320627 29670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:23:23.320904 29670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:23:23.320938 29670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:23:23.324873 29670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:23:23.324922 29670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:23:23.324939 29670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:23:23.324956 29670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:23:23.324971 29670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:23:23.329783 29670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:44385
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:44385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:23:23.330950 29670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:23:23.332293 29670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:23:23.341547 29680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:23.341627 29678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:23.341789 29670 server_base.cc:1034] running on GCE node
W20260807 06:23:23.341612 29676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:23.342433 29670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:23:23.343596 29670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:23:23.344797 29670 hybrid_clock.cc:648] HybridClock initialized: now 1786083803344765 us; error 45 us; skew 500 ppm
I20260807 06:23:23.346925 29670 webserver.cc:460] Webserver started at http://127.15.254.62:33587/ using document root <none> and password file <none>
I20260807 06:23:23.347632 29670 fs_manager.cc:359] Metadata directory not provided
I20260807 06:23:23.347730 29670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:23:23.347985 29670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:23:23.350210 29670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "5b31d0ca59e6485b88954acce6fc25a6"
format_stamp: "Formatted at 2026-08-07 06:23:23 on dist-test-slave-44d2"
server_key: "591372f5ff931144e18f887e08cb2508"
server_key_iv: "b501e9a2aded16817699032028c17f6c"
server_key_version: "encryptionkey@0"
I20260807 06:23:23.350800 29670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "5b31d0ca59e6485b88954acce6fc25a6"
format_stamp: "Formatted at 2026-08-07 06:23:23 on dist-test-slave-44d2"
server_key: "591372f5ff931144e18f887e08cb2508"
server_key_iv: "b501e9a2aded16817699032028c17f6c"
server_key_version: "encryptionkey@0"
I20260807 06:23:23.354738 29670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260807 06:23:23.357401 29686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:23.358786 29670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:23:23.358978 29670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "5b31d0ca59e6485b88954acce6fc25a6"
format_stamp: "Formatted at 2026-08-07 06:23:23 on dist-test-slave-44d2"
server_key: "591372f5ff931144e18f887e08cb2508"
server_key_iv: "b501e9a2aded16817699032028c17f6c"
server_key_version: "encryptionkey@0"
I20260807 06:23:23.359115 29670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:23:23.391364 29670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:23:23.392248 29670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:23:23.392514 29670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:23:23.400427 29670 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:44385
I20260807 06:23:23.400437 29756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:44385 every 8 connection(s)
I20260807 06:23:23.401589 29670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:23:23.405045 29758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:23:23.408471 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 29670
I20260807 06:23:23.408663 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:23:23.409003 16376 external_mini_cluster.cc:1428] Setting key 733958dfd5b93b6ecba5a25422e10f22
I20260807 06:23:23.411355 29758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b31d0ca59e6485b88954acce6fc25a6: Bootstrap starting.
I20260807 06:23:23.413753 29758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b31d0ca59e6485b88954acce6fc25a6: Neither blocks nor log segments found. Creating new log.
I20260807 06:23:23.414538 29758 log.cc:824] T 00000000000000000000000000000000 P 5b31d0ca59e6485b88954acce6fc25a6: Log is configured to *not* fsync() on all Append() calls
I20260807 06:23:23.416644 29758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b31d0ca59e6485b88954acce6fc25a6: No bootstrap required, opened a new log
I20260807 06:23:23.419380 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:23.410528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40918 (local address 127.15.254.62:44385)
0807 06:23:23.411295 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:23.411312 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:23.411350 (+ 38us) server_negotiation.cc:407] Connection header received
0807 06:23:23.412046 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:23.412064 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:23.412545 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:23.412929 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:23.413389 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:23.414463 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:23.415743 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:23.416275 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:23.416462 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:23.417528 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:23.417534 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:23.417548 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:23:23.417996 (+ 448us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:23.418062 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:23:23.418084 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:23.418210 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:23.418369 (+ 159us) server_negotiation.cc:299] Negotiation successful
0807 06:23:23.418640 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":147,"threads_started":1}
I20260807 06:23:23.420641 29758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b31d0ca59e6485b88954acce6fc25a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b31d0ca59e6485b88954acce6fc25a6" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 44385 } }
I20260807 06:23:23.420895 29758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b31d0ca59e6485b88954acce6fc25a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:23:23.420976 29758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b31d0ca59e6485b88954acce6fc25a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b31d0ca59e6485b88954acce6fc25a6, State: Initialized, Role: FOLLOWER
I20260807 06:23:23.421454 29758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b31d0ca59e6485b88954acce6fc25a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b31d0ca59e6485b88954acce6fc25a6" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 44385 } }
I20260807 06:23:23.421626 29758 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b31d0ca59e6485b88954acce6fc25a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:23:23.421690 29758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b31d0ca59e6485b88954acce6fc25a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:23:23.421784 29758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b31d0ca59e6485b88954acce6fc25a6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:23:23.422720 29758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b31d0ca59e6485b88954acce6fc25a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b31d0ca59e6485b88954acce6fc25a6" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 44385 } }
I20260807 06:23:23.423063 29758 leader_election.cc:302] T 00000000000000000000000000000000 P 5b31d0ca59e6485b88954acce6fc25a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b31d0ca59e6485b88954acce6fc25a6; no voters:
I20260807 06:23:23.423629 29758 leader_election.cc:288] T 00000000000000000000000000000000 P 5b31d0ca59e6485b88954acce6fc25a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:23:23.423795 29764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b31d0ca59e6485b88954acce6fc25a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:23:23.424060 29764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b31d0ca59e6485b88954acce6fc25a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5b31d0ca59e6485b88954acce6fc25a6, State: Running, Role: LEADER
I20260807 06:23:23.424463 29764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b31d0ca59e6485b88954acce6fc25a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b31d0ca59e6485b88954acce6fc25a6" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 44385 } }
I20260807 06:23:23.425032 29758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b31d0ca59e6485b88954acce6fc25a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:23:23.426378 29767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b31d0ca59e6485b88954acce6fc25a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b31d0ca59e6485b88954acce6fc25a6. Latest consensus state: current_term: 1 leader_uuid: "5b31d0ca59e6485b88954acce6fc25a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b31d0ca59e6485b88954acce6fc25a6" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 44385 } } }
I20260807 06:23:23.426430 29766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b31d0ca59e6485b88954acce6fc25a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b31d0ca59e6485b88954acce6fc25a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b31d0ca59e6485b88954acce6fc25a6" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 44385 } } }
I20260807 06:23:23.426608 29766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b31d0ca59e6485b88954acce6fc25a6 [sys.catalog]: This master's current role is: LEADER
I20260807 06:23:23.426911 29767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b31d0ca59e6485b88954acce6fc25a6 [sys.catalog]: This master's current role is: LEADER
I20260807 06:23:23.427268 29778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:23:23.430480 29778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:23:23.436070 29778 catalog_manager.cc:1347] Generated new cluster ID: df4c17d234194496aae269086f6b11c3
I20260807 06:23:23.436139 29778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:23:23.448881 29778 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:23:23.449798 29778 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:23:23.455468 29778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b31d0ca59e6485b88954acce6fc25a6: Generated new TSK 0
I20260807 06:23:23.456166 29778 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:23:23.465384 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:0
--local_ip_for_outbound_sockets=127.15.254.1
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:44385
--builtin_ntp_servers=127.15.254.20:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260807 06:23:23.585086 29794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:23:23.585400 29794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:23:23.585476 29794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:23:23.589663 29794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:23:23.589751 29794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:23:23.589839 29794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:23:23.594928 29794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:44385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:23:23.596143 29794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:23:23.597671 29794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:23:23.607672 29801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:23.607676 29800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:23.607686 29803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:23.608000 29794 server_base.cc:1034] running on GCE node
I20260807 06:23:23.608573 29794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:23:23.609257 29794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:23:23.610484 29794 hybrid_clock.cc:648] HybridClock initialized: now 1786083803610455 us; error 53 us; skew 500 ppm
I20260807 06:23:23.612447 29794 webserver.cc:460] Webserver started at http://127.15.254.1:46577/ using document root <none> and password file <none>
I20260807 06:23:23.613175 29794 fs_manager.cc:359] Metadata directory not provided
I20260807 06:23:23.613281 29794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:23:23.613567 29794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:23:23.615751 29794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/data/instance:
uuid: "8f6fdae218a9489586d44ce2db6a3279"
format_stamp: "Formatted at 2026-08-07 06:23:23 on dist-test-slave-44d2"
server_key: "9cdf7dce164a4e65f0d01eb30ae0f640"
server_key_iv: "99febd6ea68d1f69c6a2329cc3c1314f"
server_key_version: "encryptionkey@0"
I20260807 06:23:23.616377 29794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance:
uuid: "8f6fdae218a9489586d44ce2db6a3279"
format_stamp: "Formatted at 2026-08-07 06:23:23 on dist-test-slave-44d2"
server_key: "9cdf7dce164a4e65f0d01eb30ae0f640"
server_key_iv: "99febd6ea68d1f69c6a2329cc3c1314f"
server_key_version: "encryptionkey@0"
I20260807 06:23:23.620335 29794 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 06:23:23.623111 29815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:23.624397 29794 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 06:23:23.624591 29794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "8f6fdae218a9489586d44ce2db6a3279"
format_stamp: "Formatted at 2026-08-07 06:23:23 on dist-test-slave-44d2"
server_key: "9cdf7dce164a4e65f0d01eb30ae0f640"
server_key_iv: "99febd6ea68d1f69c6a2329cc3c1314f"
server_key_version: "encryptionkey@0"
I20260807 06:23:23.624722 29794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:23:23.637960 29794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:23:23.638643 29794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:23:23.638846 29794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:23:23.639447 29794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:23:23.640441 29794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:23:23.640545 29794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:23.640635 29794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:23:23.640686 29794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:23.650300 29794 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:38923
I20260807 06:23:23.650332 29975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:38923 every 8 connection(s)
I20260807 06:23:23.651234 29794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:23:23.653558 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 29794
I20260807 06:23:23.653697 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance
I20260807 06:23:23.654096 16376 external_mini_cluster.cc:1428] Setting key b6f557e43c60644fdafa349920cadc6a
I20260807 06:23:23.656235 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:0
--local_ip_for_outbound_sockets=127.15.254.2
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:44385
--builtin_ntp_servers=127.15.254.20:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260807 06:23:23.662436 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:23.653108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:47997 (local address 127.15.254.62:44385)
0807 06:23:23.653249 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:23.653252 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:23.654662 (+ 1410us) server_negotiation.cc:407] Connection header received
0807 06:23:23.656435 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:23.656441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:23.656531 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:23.656635 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:23.657578 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:23.658368 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:23.660185 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:23.660415 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:23.660551 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:23.661896 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:23.661899 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:23.661901 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:23:23.661914 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:23.661934 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:23:23.661942 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:23.662012 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:23.662222 (+ 210us) server_negotiation.cc:299] Negotiation successful
0807 06:23:23.662312 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 06:23:23.663739 29984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:23.653596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:44385 (local address 127.15.254.1:47997)
0807 06:23:23.654456 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:23.654533 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:23.656167 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:23.656798 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:23.656805 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:23.656891 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:23.657391 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:23.657404 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:23.658528 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:23.658534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:23.660048 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:23.660063 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:23.660607 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:23.660613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:23.660854 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:23.661543 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:23:23.661546 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:23.661608 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:23:23.661706 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:23:23.662039 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:23:23.662053 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:23:23.662063 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 06:23:23.662217 (+ 154us) client_negotiation.cc:240] Negotiation successful
0807 06:23:23.662577 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":167,"threads_started":1}
I20260807 06:23:23.665102 29978 heartbeater.cc:347] Connected to a master server at 127.15.254.62:44385
I20260807 06:23:23.665347 29978 heartbeater.cc:464] Registering TS with master...
I20260807 06:23:23.665913 29978 heartbeater.cc:511] Master 127.15.254.62:44385 requested a full tablet report, sending...
I20260807 06:23:23.667464 29710 ts_manager.cc:194] Registered new tserver with Master: 8f6fdae218a9489586d44ce2db6a3279 (127.15.254.1:38923)
I20260807 06:23:23.668778 29710 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.254.1:47997
W20260807 06:23:23.776185 29987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:23:23.776499 29987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:23:23.776618 29987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:23:23.780776 29987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:23:23.780874 29987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:23:23.780975 29987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:23:23.785768 29987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:44385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:23:23.786963 29987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:23:23.788430 29987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:23:23.798482 29996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:23.798616 29987 server_base.cc:1034] running on GCE node
W20260807 06:23:23.798482 29997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:23.798501 29999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:23.799178 29987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:23:23.799875 29987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:23:23.801065 29987 hybrid_clock.cc:648] HybridClock initialized: now 1786083803801031 us; error 48 us; skew 500 ppm
I20260807 06:23:23.802901 29987 webserver.cc:460] Webserver started at http://127.15.254.2:41265/ using document root <none> and password file <none>
I20260807 06:23:23.803508 29987 fs_manager.cc:359] Metadata directory not provided
I20260807 06:23:23.803608 29987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:23:23.803855 29987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:23:23.805984 29987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/data/instance:
uuid: "8192cff63a6e44d78cdafb3a91f11b0d"
format_stamp: "Formatted at 2026-08-07 06:23:23 on dist-test-slave-44d2"
server_key: "e46b877b9e8d67c7a6bdf991ca58459b"
server_key_iv: "225de6589a735a7804ef49f7180a4c09"
server_key_version: "encryptionkey@0"
I20260807 06:23:23.806599 29987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance:
uuid: "8192cff63a6e44d78cdafb3a91f11b0d"
format_stamp: "Formatted at 2026-08-07 06:23:23 on dist-test-slave-44d2"
server_key: "e46b877b9e8d67c7a6bdf991ca58459b"
server_key_iv: "225de6589a735a7804ef49f7180a4c09"
server_key_version: "encryptionkey@0"
I20260807 06:23:23.810405 29987 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:23:23.813094 30006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:23.814358 29987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 06:23:23.814523 29987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "8192cff63a6e44d78cdafb3a91f11b0d"
format_stamp: "Formatted at 2026-08-07 06:23:23 on dist-test-slave-44d2"
server_key: "e46b877b9e8d67c7a6bdf991ca58459b"
server_key_iv: "225de6589a735a7804ef49f7180a4c09"
server_key_version: "encryptionkey@0"
I20260807 06:23:23.814651 29987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:23:23.832688 29987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:23:23.833477 29987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:23:23.833730 29987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:23:23.834581 29987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:23:23.835680 29987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:23:23.835727 29987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:23.835801 29987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:23:23.835840 29987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:23.845355 29987 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:40709
I20260807 06:23:23.845389 30156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:40709 every 8 connection(s)
I20260807 06:23:23.846310 29987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:23:23.854737 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 29987
I20260807 06:23:23.854871 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance
I20260807 06:23:23.855248 16376 external_mini_cluster.cc:1428] Setting key ce41ad51b4a74ded8c97d3bbe0726fb1
I20260807 06:23:23.856017 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:23.848259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:51621 (local address 127.15.254.62:44385)
0807 06:23:23.848420 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:23.848424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:23.849584 (+ 1160us) server_negotiation.cc:407] Connection header received
0807 06:23:23.850997 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:23.851000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:23.851049 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:23.851127 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:23.852047 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:23.852685 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:23.853860 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:23.854100 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:23.854186 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:23.855460 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:23.855463 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:23.855465 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:23:23.855479 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:23.855498 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:23:23.855506 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:23.855590 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:23.855787 (+ 197us) server_negotiation.cc:299] Negotiation successful
0807 06:23:23.855857 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 06:23:23.857090 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:23.848707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:44385 (local address 127.15.254.2:51621)
0807 06:23:23.849414 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:23.849457 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:23.850786 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:23.851302 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:23.851309 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:23.851396 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:23.851885 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:23.851898 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:23.852804 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:23.852808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:23.853734 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:23.853744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:23.854242 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:23.854245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:23.854388 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:23.855116 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:23:23.855119 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:23.855179 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:23:23.855278 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:23:23.855601 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:23:23.855612 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:23:23.855618 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 06:23:23.855757 (+ 139us) client_negotiation.cc:240] Negotiation successful
0807 06:23:23.856037 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":123,"threads_started":1}
I20260807 06:23:23.857523 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:0
--local_ip_for_outbound_sockets=127.15.254.3
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:44385
--builtin_ntp_servers=127.15.254.20:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260807 06:23:23.858327 30158 heartbeater.cc:347] Connected to a master server at 127.15.254.62:44385
I20260807 06:23:23.858592 30158 heartbeater.cc:464] Registering TS with master...
I20260807 06:23:23.859253 30158 heartbeater.cc:511] Master 127.15.254.62:44385 requested a full tablet report, sending...
I20260807 06:23:23.860554 29710 ts_manager.cc:194] Registered new tserver with Master: 8192cff63a6e44d78cdafb3a91f11b0d (127.15.254.2:40709)
I20260807 06:23:23.861333 29710 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.254.2:51621
W20260807 06:23:23.980053 30162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:23:23.980357 30162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:23:23.980399 30162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:23:23.984874 30162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:23:23.985046 30162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:23:23.985199 30162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:23:23.990394 30162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:44385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:23:23.991580 30162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:23:23.993121 30162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:23:24.003738 30175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:24.003767 30172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:24.003741 30168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:24.003990 30162 server_base.cc:1034] running on GCE node
I20260807 06:23:24.004876 30162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:23:24.005621 30162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:23:24.006794 30162 hybrid_clock.cc:648] HybridClock initialized: now 1786083804006760 us; error 49 us; skew 500 ppm
I20260807 06:23:24.008996 30162 webserver.cc:460] Webserver started at http://127.15.254.3:46361/ using document root <none> and password file <none>
I20260807 06:23:24.009641 30162 fs_manager.cc:359] Metadata directory not provided
I20260807 06:23:24.009711 30162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:23:24.009982 30162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:23:24.012224 30162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/data/instance:
uuid: "28f6af3e698e4b64ad0282673bd263f5"
format_stamp: "Formatted at 2026-08-07 06:23:24 on dist-test-slave-44d2"
server_key: "733589fab258716078dcdaa4f548d8bd"
server_key_iv: "da6468dcd937b2cf59cd0a7c7fbb58bd"
server_key_version: "encryptionkey@0"
I20260807 06:23:24.012923 30162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance:
uuid: "28f6af3e698e4b64ad0282673bd263f5"
format_stamp: "Formatted at 2026-08-07 06:23:24 on dist-test-slave-44d2"
server_key: "733589fab258716078dcdaa4f548d8bd"
server_key_iv: "da6468dcd937b2cf59cd0a7c7fbb58bd"
server_key_version: "encryptionkey@0"
I20260807 06:23:24.018576 30162 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 06:23:24.021530 30182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:24.022959 30162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:23:24.023123 30162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "28f6af3e698e4b64ad0282673bd263f5"
format_stamp: "Formatted at 2026-08-07 06:23:24 on dist-test-slave-44d2"
server_key: "733589fab258716078dcdaa4f548d8bd"
server_key_iv: "da6468dcd937b2cf59cd0a7c7fbb58bd"
server_key_version: "encryptionkey@0"
I20260807 06:23:24.023252 30162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:23:24.038480 30162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:23:24.039202 30162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:23:24.039412 30162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:23:24.040053 30162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:23:24.041139 30162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:23:24.041214 30162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:24.041287 30162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:23:24.041340 30162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:24.050627 30162 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:41859
I20260807 06:23:24.050673 30337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:41859 every 8 connection(s)
I20260807 06:23:24.051551 30162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:23:24.056345 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 30162
I20260807 06:23:24.056478 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance
I20260807 06:23:24.056818 16376 external_mini_cluster.cc:1428] Setting key 591fa3d098725b4a52f6f08edf62f297
I20260807 06:23:24.061735 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:24.053520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:50267 (local address 127.15.254.62:44385)
0807 06:23:24.053676 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:24.053680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:24.054798 (+ 1118us) server_negotiation.cc:407] Connection header received
0807 06:23:24.056308 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:24.056311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:24.056373 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:24.056467 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:24.057711 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:24.058362 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:24.059833 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:24.060067 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:24.060151 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:24.061348 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:24.061351 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:24.061353 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:23:24.061364 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:24.061381 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:23:24.061389 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:24.061447 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:24.061565 (+ 118us) server_negotiation.cc:299] Negotiation successful
0807 06:23:24.061618 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 06:23:24.063047 30341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:24.053927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:44385 (local address 127.15.254.3:50267)
0807 06:23:24.054643 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:24.054684 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:24.056060 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:24.056750 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:24.056761 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:24.056874 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:24.057521 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:24.057536 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:24.058516 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:24.058520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:24.059720 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:24.059731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:24.060207 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:24.060210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:24.060353 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:24.061096 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:23:24.061098 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:24.061142 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:23:24.061211 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:23:24.061447 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:23:24.061451 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:23:24.061455 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 06:23:24.061545 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 06:23:24.061805 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":125,"threads_started":1}
I20260807 06:23:24.064365 30338 heartbeater.cc:347] Connected to a master server at 127.15.254.62:44385
I20260807 06:23:24.064639 30338 heartbeater.cc:464] Registering TS with master...
I20260807 06:23:24.065219 30338 heartbeater.cc:511] Master 127.15.254.62:44385 requested a full tablet report, sending...
I20260807 06:23:24.066332 29710 ts_manager.cc:194] Registered new tserver with Master: 28f6af3e698e4b64ad0282673bd263f5 (127.15.254.3:41859)
I20260807 06:23:24.066955 29710 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.254.3:50267
I20260807 06:23:24.071344 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 06:23:24.074384 29762 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:24.073416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40934 (local address 127.15.254.62:44385)
0807 06:23:24.073546 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:24.073550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:24.073806 (+ 256us) server_negotiation.cc:407] Connection header received
0807 06:23:24.073913 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:24.073915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:24.073953 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:24.074022 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:24.074286 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40934: BlockingRecv error: recv got EOF from 127.0.0.1:40934 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 06:23:24.074901 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 29794
I20260807 06:23:24.089392 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 29987
I20260807 06:23:24.111064 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 30162
I20260807 06:23:24.123539 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 29670
2026-08-07T06:23:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (962 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-07T06:23:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:23:24Z Disabled control of system clock
I20260807 06:23:24.163272 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:43741
--webserver_interface=127.15.254.62
--webserver_port=0
--builtin_ntp_servers=127.15.254.20:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:43741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 06:23:24.278992 30351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:23:24.279253 30351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:23:24.279289 30351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:23:24.283200 30351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:23:24.283263 30351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:23:24.283279 30351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:23:24.283295 30351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:23:24.283311 30351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:23:24.288005 30351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:43741
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:43741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:23:24.289196 30351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:23:24.290637 30351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:23:24.299767 30361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:24.299753 30364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:24.299934 30351 server_base.cc:1034] running on GCE node
W20260807 06:23:24.300169 30357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:24.300832 30351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:23:24.302032 30351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:23:24.303243 30351 hybrid_clock.cc:648] HybridClock initialized: now 1786083804303215 us; error 47 us; skew 500 ppm
I20260807 06:23:24.305392 30351 webserver.cc:460] Webserver started at http://127.15.254.62:46389/ using document root <none> and password file <none>
I20260807 06:23:24.306046 30351 fs_manager.cc:359] Metadata directory not provided
I20260807 06:23:24.306156 30351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:23:24.306411 30351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:23:24.308634 30351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "99acc6c4e4ca46a59ab1cf0ed52967e0"
format_stamp: "Formatted at 2026-08-07 06:23:24 on dist-test-slave-44d2"
server_key: "d476ad3d7396bddf786e3f4876736552"
server_key_iv: "3e4ee3aa0a1b316b2d63558006c241cc"
server_key_version: "encryptionkey@0"
I20260807 06:23:24.309265 30351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "99acc6c4e4ca46a59ab1cf0ed52967e0"
format_stamp: "Formatted at 2026-08-07 06:23:24 on dist-test-slave-44d2"
server_key: "d476ad3d7396bddf786e3f4876736552"
server_key_iv: "3e4ee3aa0a1b316b2d63558006c241cc"
server_key_version: "encryptionkey@0"
I20260807 06:23:24.313259 30351 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:23:24.315876 30374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:24.317251 30351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:23:24.317406 30351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "99acc6c4e4ca46a59ab1cf0ed52967e0"
format_stamp: "Formatted at 2026-08-07 06:23:24 on dist-test-slave-44d2"
server_key: "d476ad3d7396bddf786e3f4876736552"
server_key_iv: "3e4ee3aa0a1b316b2d63558006c241cc"
server_key_version: "encryptionkey@0"
I20260807 06:23:24.317531 30351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:23:24.322388 30351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:23:24.323036 30351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:23:24.323241 30351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:23:24.330636 30351 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:43741
I20260807 06:23:24.330708 30445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:43741 every 8 connection(s)
I20260807 06:23:24.331807 30351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:23:24.335589 30446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:23:24.339758 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 30351
I20260807 06:23:24.339917 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:23:24.340224 16376 external_mini_cluster.cc:1428] Setting key fe5c871759bc97f5524415625c594f78
I20260807 06:23:24.341948 30446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99acc6c4e4ca46a59ab1cf0ed52967e0: Bootstrap starting.
I20260807 06:23:24.344372 30446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99acc6c4e4ca46a59ab1cf0ed52967e0: Neither blocks nor log segments found. Creating new log.
I20260807 06:23:24.345247 30446 log.cc:824] T 00000000000000000000000000000000 P 99acc6c4e4ca46a59ab1cf0ed52967e0: Log is configured to *not* fsync() on all Append() calls
I20260807 06:23:24.346186 30450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:24.341748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42696 (local address 127.15.254.62:43741)
0807 06:23:24.342513 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:24.342529 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:24.342567 (+ 38us) server_negotiation.cc:407] Connection header received
0807 06:23:24.343232 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:24.343251 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:24.343621 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:24.343971 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:24.344857 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:24.344862 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:24.344875 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:23:24.344886 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:24.344959 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:23:24.344977 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:24.345110 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:24.345213 (+ 103us) server_negotiation.cc:299] Negotiation successful
0807 06:23:24.345496 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":147,"threads_started":1}
I20260807 06:23:24.347566 30446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99acc6c4e4ca46a59ab1cf0ed52967e0: No bootstrap required, opened a new log
I20260807 06:23:24.351155 30446 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99acc6c4e4ca46a59ab1cf0ed52967e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99acc6c4e4ca46a59ab1cf0ed52967e0" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43741 } }
I20260807 06:23:24.351406 30446 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99acc6c4e4ca46a59ab1cf0ed52967e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:23:24.351498 30446 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99acc6c4e4ca46a59ab1cf0ed52967e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99acc6c4e4ca46a59ab1cf0ed52967e0, State: Initialized, Role: FOLLOWER
I20260807 06:23:24.351948 30446 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99acc6c4e4ca46a59ab1cf0ed52967e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99acc6c4e4ca46a59ab1cf0ed52967e0" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43741 } }
I20260807 06:23:24.352103 30446 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99acc6c4e4ca46a59ab1cf0ed52967e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:23:24.352192 30446 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99acc6c4e4ca46a59ab1cf0ed52967e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:23:24.352324 30446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99acc6c4e4ca46a59ab1cf0ed52967e0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:23:24.353444 30446 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99acc6c4e4ca46a59ab1cf0ed52967e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99acc6c4e4ca46a59ab1cf0ed52967e0" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43741 } }
I20260807 06:23:24.353807 30446 leader_election.cc:302] T 00000000000000000000000000000000 P 99acc6c4e4ca46a59ab1cf0ed52967e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99acc6c4e4ca46a59ab1cf0ed52967e0; no voters:
I20260807 06:23:24.354142 30446 leader_election.cc:288] T 00000000000000000000000000000000 P 99acc6c4e4ca46a59ab1cf0ed52967e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:23:24.354254 30453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99acc6c4e4ca46a59ab1cf0ed52967e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:23:24.354512 30453 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99acc6c4e4ca46a59ab1cf0ed52967e0 [term 1 LEADER]: Becoming Leader. State: Replica: 99acc6c4e4ca46a59ab1cf0ed52967e0, State: Running, Role: LEADER
I20260807 06:23:24.354866 30453 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99acc6c4e4ca46a59ab1cf0ed52967e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99acc6c4e4ca46a59ab1cf0ed52967e0" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43741 } }
I20260807 06:23:24.355434 30446 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99acc6c4e4ca46a59ab1cf0ed52967e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:23:24.356535 30454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99acc6c4e4ca46a59ab1cf0ed52967e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99acc6c4e4ca46a59ab1cf0ed52967e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99acc6c4e4ca46a59ab1cf0ed52967e0" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43741 } } }
I20260807 06:23:24.356678 30454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99acc6c4e4ca46a59ab1cf0ed52967e0 [sys.catalog]: This master's current role is: LEADER
I20260807 06:23:24.357003 30455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99acc6c4e4ca46a59ab1cf0ed52967e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99acc6c4e4ca46a59ab1cf0ed52967e0. Latest consensus state: current_term: 1 leader_uuid: "99acc6c4e4ca46a59ab1cf0ed52967e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99acc6c4e4ca46a59ab1cf0ed52967e0" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43741 } } }
I20260807 06:23:24.357079 30466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:23:24.357138 30455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99acc6c4e4ca46a59ab1cf0ed52967e0 [sys.catalog]: This master's current role is: LEADER
I20260807 06:23:24.360261 30466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:23:24.365866 30466 catalog_manager.cc:1347] Generated new cluster ID: 3914905164d44f16a1b6ec12b8fbe231
I20260807 06:23:24.365938 30466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:23:24.395671 30466 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:23:24.396101 30466 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:23:24.410234 30466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99acc6c4e4ca46a59ab1cf0ed52967e0: Generated new TSK 0
I20260807 06:23:24.411057 30466 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:23:24.417196 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:0
--local_ip_for_outbound_sockets=127.15.254.1
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:43741
--builtin_ntp_servers=127.15.254.20:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 06:23:24.538906 30484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:23:24.539172 30484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:23:24.539244 30484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:23:24.543306 30484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:23:24.543371 30484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:23:24.543483 30484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:23:24.548384 30484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:43741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:23:24.549499 30484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:23:24.550884 30484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:23:24.560864 30496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:24.560866 30491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:24.560971 30492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:24.561122 30484 server_base.cc:1034] running on GCE node
I20260807 06:23:24.561635 30484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:23:24.562387 30484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:23:24.563575 30484 hybrid_clock.cc:648] HybridClock initialized: now 1786083804563555 us; error 32 us; skew 500 ppm
I20260807 06:23:24.565636 30484 webserver.cc:460] Webserver started at http://127.15.254.1:41049/ using document root <none> and password file <none>
I20260807 06:23:24.566294 30484 fs_manager.cc:359] Metadata directory not provided
I20260807 06:23:24.566406 30484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:23:24.566669 30484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:23:24.568993 30484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/data/instance:
uuid: "b9858261895d410da504c514d6f5743e"
format_stamp: "Formatted at 2026-08-07 06:23:24 on dist-test-slave-44d2"
server_key: "7e457f1a18a1d16a9abd9e66e738e56b"
server_key_iv: "a85bbc34ccebbd93aabae68a49444f73"
server_key_version: "encryptionkey@0"
I20260807 06:23:24.569609 30484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance:
uuid: "b9858261895d410da504c514d6f5743e"
format_stamp: "Formatted at 2026-08-07 06:23:24 on dist-test-slave-44d2"
server_key: "7e457f1a18a1d16a9abd9e66e738e56b"
server_key_iv: "a85bbc34ccebbd93aabae68a49444f73"
server_key_version: "encryptionkey@0"
I20260807 06:23:24.573894 30484 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:23:24.577131 30503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:24.578528 30484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:23:24.578677 30484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "b9858261895d410da504c514d6f5743e"
format_stamp: "Formatted at 2026-08-07 06:23:24 on dist-test-slave-44d2"
server_key: "7e457f1a18a1d16a9abd9e66e738e56b"
server_key_iv: "a85bbc34ccebbd93aabae68a49444f73"
server_key_version: "encryptionkey@0"
I20260807 06:23:24.578805 30484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:23:24.584079 30484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:23:24.584824 30484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:23:24.585045 30484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:23:24.585704 30484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:23:24.586745 30484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:23:24.586796 30484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:24.586884 30484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:23:24.586918 30484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:24.596910 30484 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:39701
I20260807 06:23:24.596941 30650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:39701 every 8 connection(s)
I20260807 06:23:24.597934 30484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:23:24.604135 30450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:24.599880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:46229 (local address 127.15.254.62:43741)
0807 06:23:24.600053 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:24.600059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:24.601169 (+ 1110us) server_negotiation.cc:407] Connection header received
0807 06:23:24.602570 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:24.602574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:24.602611 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:24.602688 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:24.603769 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:24.603772 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:24.603774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:23:24.603785 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:24.603805 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:23:24.603813 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:24.603886 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:24.603989 (+ 103us) server_negotiation.cc:299] Negotiation successful
0807 06:23:24.604028 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 06:23:24.605068 30653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:24.600307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:43741 (local address 127.15.254.1:46229)
0807 06:23:24.601008 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:24.601052 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:24.602336 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:24.602863 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:24.602869 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:24.602961 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:24.603498 (+ 537us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:23:24.603500 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:24.603542 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:23:24.603617 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:23:24.603873 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:23:24.603877 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:23:24.603880 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 06:23:24.604005 (+ 125us) client_negotiation.cc:240] Negotiation successful
0807 06:23:24.604237 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":130,"threads_started":1}
I20260807 06:23:24.605391 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 30484
I20260807 06:23:24.605530 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance
I20260807 06:23:24.605911 16376 external_mini_cluster.cc:1428] Setting key 546f5530328bfb40b097b44ccd12cf41
I20260807 06:23:24.606439 30651 heartbeater.cc:347] Connected to a master server at 127.15.254.62:43741
I20260807 06:23:24.606711 30651 heartbeater.cc:464] Registering TS with master...
I20260807 06:23:24.607122 30651 heartbeater.cc:511] Master 127.15.254.62:43741 requested a full tablet report, sending...
I20260807 06:23:24.608207 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:0
--local_ip_for_outbound_sockets=127.15.254.2
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:43741
--builtin_ntp_servers=127.15.254.20:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260807 06:23:24.608769 30399 ts_manager.cc:194] Registered new tserver with Master: b9858261895d410da504c514d6f5743e (127.15.254.1:39701)
W20260807 06:23:24.729342 30655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:23:24.729622 30655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:23:24.729663 30655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:23:24.733629 30655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:23:24.733675 30655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:23:24.733731 30655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:23:24.738387 30655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:43741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:23:24.739454 30655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:23:24.740875 30655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:23:24.750954 30665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:24.750965 30662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:24.751004 30655 server_base.cc:1034] running on GCE node
W20260807 06:23:24.751053 30663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:24.751624 30655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:23:24.752285 30655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:23:24.753455 30655 hybrid_clock.cc:648] HybridClock initialized: now 1786083804753445 us; error 55 us; skew 500 ppm
I20260807 06:23:24.755704 30655 webserver.cc:460] Webserver started at http://127.15.254.2:38907/ using document root <none> and password file <none>
I20260807 06:23:24.756382 30655 fs_manager.cc:359] Metadata directory not provided
I20260807 06:23:24.756456 30655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:23:24.756781 30655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:23:24.758948 30655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/data/instance:
uuid: "6a536839586449ec927c4d83f55771f7"
format_stamp: "Formatted at 2026-08-07 06:23:24 on dist-test-slave-44d2"
server_key: "c3214008624cbc37255f9746bd2b1642"
server_key_iv: "3a04bc3d81235a98d098026d51733ac2"
server_key_version: "encryptionkey@0"
I20260807 06:23:24.759496 30655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance:
uuid: "6a536839586449ec927c4d83f55771f7"
format_stamp: "Formatted at 2026-08-07 06:23:24 on dist-test-slave-44d2"
server_key: "c3214008624cbc37255f9746bd2b1642"
server_key_iv: "3a04bc3d81235a98d098026d51733ac2"
server_key_version: "encryptionkey@0"
I20260807 06:23:24.763269 30655 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260807 06:23:24.766153 30672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:24.767520 30655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 06:23:24.767652 30655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "6a536839586449ec927c4d83f55771f7"
format_stamp: "Formatted at 2026-08-07 06:23:24 on dist-test-slave-44d2"
server_key: "c3214008624cbc37255f9746bd2b1642"
server_key_iv: "3a04bc3d81235a98d098026d51733ac2"
server_key_version: "encryptionkey@0"
I20260807 06:23:24.767774 30655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:23:24.772941 30655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:23:24.773622 30655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:23:24.773820 30655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:23:24.774410 30655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:23:24.775393 30655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:23:24.775444 30655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:24.775512 30655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:23:24.775552 30655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:24.785346 30655 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:34119
I20260807 06:23:24.785442 30830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:34119 every 8 connection(s)
I20260807 06:23:24.786373 30655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:23:24.793226 30450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:24.788375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:49445 (local address 127.15.254.62:43741)
0807 06:23:24.788594 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:24.788598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:24.789725 (+ 1127us) server_negotiation.cc:407] Connection header received
0807 06:23:24.791280 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:24.791283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:24.791321 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:24.791398 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:24.792783 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:24.792786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:24.792788 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:23:24.792800 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:24.792819 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:23:24.792833 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:24.792920 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:24.793076 (+ 156us) server_negotiation.cc:299] Negotiation successful
0807 06:23:24.793114 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 06:23:24.794183 30833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:24.788850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:43741 (local address 127.15.254.2:49445)
0807 06:23:24.789579 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:24.789621 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:24.791065 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:24.791588 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:24.791595 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:24.791690 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:24.792430 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:23:24.792432 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:24.792530 (+ 98us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:23:24.792637 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:23:24.792937 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:23:24.792945 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:23:24.792950 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 06:23:24.793101 (+ 151us) client_negotiation.cc:240] Negotiation successful
0807 06:23:24.793307 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":119,"threads_started":1}
I20260807 06:23:24.795385 30831 heartbeater.cc:347] Connected to a master server at 127.15.254.62:43741
I20260807 06:23:24.795647 30831 heartbeater.cc:464] Registering TS with master...
I20260807 06:23:24.795890 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 30655
I20260807 06:23:24.796025 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance
I20260807 06:23:24.796182 30831 heartbeater.cc:511] Master 127.15.254.62:43741 requested a full tablet report, sending...
I20260807 06:23:24.796381 16376 external_mini_cluster.cc:1428] Setting key e90b6a224866961d0f75bd6c97013c68
I20260807 06:23:24.797362 30398 ts_manager.cc:194] Registered new tserver with Master: 6a536839586449ec927c4d83f55771f7 (127.15.254.2:34119)
I20260807 06:23:24.798512 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:0
--local_ip_for_outbound_sockets=127.15.254.3
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:43741
--builtin_ntp_servers=127.15.254.20:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 06:23:24.919380 30834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:23:24.919675 30834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:23:24.919719 30834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:23:24.924054 30834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:23:24.924127 30834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:23:24.924188 30834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:23:24.929242 30834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:43741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:23:24.930469 30834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:23:24.931931 30834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:23:24.942400 30841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:24.942495 30834 server_base.cc:1034] running on GCE node
W20260807 06:23:24.942407 30843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:24.942422 30840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:24.943105 30834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:23:24.943776 30834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:23:24.945007 30834 hybrid_clock.cc:648] HybridClock initialized: now 1786083804944967 us; error 52 us; skew 500 ppm
I20260807 06:23:24.947091 30834 webserver.cc:460] Webserver started at http://127.15.254.3:34989/ using document root <none> and password file <none>
I20260807 06:23:24.947715 30834 fs_manager.cc:359] Metadata directory not provided
I20260807 06:23:24.947817 30834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:23:24.948101 30834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:23:24.950366 30834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/data/instance:
uuid: "4074d5bed43247fdb194e5b033cd8555"
format_stamp: "Formatted at 2026-08-07 06:23:24 on dist-test-slave-44d2"
server_key: "3f53200055508698478449f59ecd9068"
server_key_iv: "27ae1a736103e0c099b3641642349324"
server_key_version: "encryptionkey@0"
I20260807 06:23:24.951910 30834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance:
uuid: "4074d5bed43247fdb194e5b033cd8555"
format_stamp: "Formatted at 2026-08-07 06:23:24 on dist-test-slave-44d2"
server_key: "3f53200055508698478449f59ecd9068"
server_key_iv: "27ae1a736103e0c099b3641642349324"
server_key_version: "encryptionkey@0"
I20260807 06:23:24.956341 30834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 06:23:24.959465 30851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:24.961055 30834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:23:24.961212 30834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "4074d5bed43247fdb194e5b033cd8555"
format_stamp: "Formatted at 2026-08-07 06:23:24 on dist-test-slave-44d2"
server_key: "3f53200055508698478449f59ecd9068"
server_key_iv: "27ae1a736103e0c099b3641642349324"
server_key_version: "encryptionkey@0"
I20260807 06:23:24.961344 30834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:23:24.966372 30834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:23:24.967064 30834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:23:24.967269 30834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:23:24.967857 30834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:23:24.968879 30834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:23:24.968963 30834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:24.969036 30834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:23:24.969092 30834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:24.978523 30834 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:36921
I20260807 06:23:24.978592 31012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:36921 every 8 connection(s)
I20260807 06:23:24.979568 30834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:23:24.986549 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 30834
I20260807 06:23:24.986708 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance
I20260807 06:23:24.987076 16376 external_mini_cluster.cc:1428] Setting key 15790a2a7f7aacb26dae63dfb4e7ba42
I20260807 06:23:24.987336 30450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:24.981687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:50039 (local address 127.15.254.62:43741)
0807 06:23:24.981857 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:24.981862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:24.983388 (+ 1526us) server_negotiation.cc:407] Connection header received
0807 06:23:24.985122 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:24.985126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:24.985179 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:24.985312 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:24.986704 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:24.986711 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:24.986718 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:23:24.986731 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:24.986770 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:23:24.986786 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:24.986880 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:24.987168 (+ 288us) server_negotiation.cc:299] Negotiation successful
0807 06:23:24.987227 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 06:23:24.988670 31017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:24.982111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:43741 (local address 127.15.254.3:50039)
0807 06:23:24.983206 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:24.983253 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:24.984881 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:24.985507 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:24.985515 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:24.985621 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:24.986305 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:23:24.986310 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:24.986376 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:23:24.986482 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:23:24.986963 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:23:24.986974 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:23:24.986981 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 06:23:24.987195 (+ 214us) client_negotiation.cc:240] Negotiation successful
0807 06:23:24.987478 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":112,"threads_started":1}
I20260807 06:23:24.989848 31013 heartbeater.cc:347] Connected to a master server at 127.15.254.62:43741
I20260807 06:23:24.990113 31013 heartbeater.cc:464] Registering TS with master...
I20260807 06:23:24.990602 31013 heartbeater.cc:511] Master 127.15.254.62:43741 requested a full tablet report, sending...
I20260807 06:23:24.991643 30398 ts_manager.cc:194] Registered new tserver with Master: 4074d5bed43247fdb194e5b033cd8555 (127.15.254.3:36921)
I20260807 06:23:25.001711 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 06:23:25.005254 30450 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:23:25.004006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42708 (local address 127.15.254.62:43741)
0807 06:23:25.004189 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:25.004194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:25.004443 (+ 249us) server_negotiation.cc:407] Connection header received
0807 06:23:25.004645 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:25.004652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:25.004691 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:25.004779 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:25.005102 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42708: BlockingRecv error: recv got EOF from 127.0.0.1:42708 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 06:23:25.005574 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 30484
I20260807 06:23:25.025035 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 30655
I20260807 06:23:25.047201 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 30834
I20260807 06:23:25.058279 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 30351
2026-08-07T06:23:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (933 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:23:25 dist-test-slave-44d2 krb5kdc[31028](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:23:25 dist-test-slave-44d2 krb5kdc[31028](info): set up 2 sockets
Aug 07 06:23:25 dist-test-slave-44d2 krb5kdc[31028](info): commencing operation
krb5kdc: starting...
W20260807 06:23:26.301819 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.214s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:23:26 dist-test-slave-44d2 krb5kdc[31028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:23:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:23:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:23:26.439353 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:43507
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:42951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:43507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:23:26.556237 31046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:23:26.556556 31046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:23:26.556638 31046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:23:26.561211 31046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:23:26.561309 31046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:23:26.561350 31046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:23:26.561389 31046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:23:26.561419 31046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:23:26.566670 31046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:42951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:43507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:43507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:23:26.567778 31046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:23:26.568878 31046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:23:26.578051 31054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:26.578051 31052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:26.578248 31046 server_base.cc:1034] running on GCE node
W20260807 06:23:26.578138 31058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:26.578907 31046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:23:26.580103 31046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:23:26.581310 31046 hybrid_clock.cc:648] HybridClock initialized: now 1786083806581277 us; error 44 us; skew 500 ppm
Aug 07 06:23:26 dist-test-slave-44d2 krb5kdc[31028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083806, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:23:26.584340 31046 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:23:26.585320 31046 webserver.cc:460] Webserver started at http://127.15.254.62:43031/ using document root <none> and password file <none>
I20260807 06:23:26.585930 31046 fs_manager.cc:359] Metadata directory not provided
I20260807 06:23:26.586031 31046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:23:26.586314 31046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:23:26.588591 31046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "70e9b1eb672b496d962893879bff3b46"
format_stamp: "Formatted at 2026-08-07 06:23:26 on dist-test-slave-44d2"
server_key: "001b9b81f034a227a2a317e91ab9a777"
server_key_iv: "f87098bb5c88e273ddc2318fcc9b3b1e"
server_key_version: "encryptionkey@0"
I20260807 06:23:26.589192 31046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "70e9b1eb672b496d962893879bff3b46"
format_stamp: "Formatted at 2026-08-07 06:23:26 on dist-test-slave-44d2"
server_key: "001b9b81f034a227a2a317e91ab9a777"
server_key_iv: "f87098bb5c88e273ddc2318fcc9b3b1e"
server_key_version: "encryptionkey@0"
I20260807 06:23:26.593036 31046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 06:23:26.595526 31069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:26.596879 31046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 06:23:26.597029 31046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "70e9b1eb672b496d962893879bff3b46"
format_stamp: "Formatted at 2026-08-07 06:23:26 on dist-test-slave-44d2"
server_key: "001b9b81f034a227a2a317e91ab9a777"
server_key_iv: "f87098bb5c88e273ddc2318fcc9b3b1e"
server_key_version: "encryptionkey@0"
I20260807 06:23:26.597154 31046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:23:26.610091 31046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:23:26.613056 31046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:23:26.613286 31046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:23:26.620507 31046 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:43507
I20260807 06:23:26.620826 31140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:43507 every 8 connection(s)
I20260807 06:23:26.621591 31046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:23:26.624723 31141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:23:26.626744 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 31046
I20260807 06:23:26.626907 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:23:26.627197 16376 external_mini_cluster.cc:1428] Setting key 2a31b1abda1e880d88893dc330938d5d
I20260807 06:23:26.632037 31141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e9b1eb672b496d962893879bff3b46: Bootstrap starting.
I20260807 06:23:26.634786 31141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70e9b1eb672b496d962893879bff3b46: Neither blocks nor log segments found. Creating new log.
I20260807 06:23:26.635540 31141 log.cc:824] T 00000000000000000000000000000000 P 70e9b1eb672b496d962893879bff3b46: Log is configured to *not* fsync() on all Append() calls
Aug 07 06:23:26 dist-test-slave-44d2 krb5kdc[31028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:23:26.638048 31141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e9b1eb672b496d962893879bff3b46: No bootstrap required, opened a new log
I20260807 06:23:26.643007 31141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70e9b1eb672b496d962893879bff3b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e9b1eb672b496d962893879bff3b46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43507 } }
I20260807 06:23:26.643285 31141 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70e9b1eb672b496d962893879bff3b46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:23:26.643334 31141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70e9b1eb672b496d962893879bff3b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e9b1eb672b496d962893879bff3b46, State: Initialized, Role: FOLLOWER
I20260807 06:23:26.643556 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:26.628728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55794 (local address 127.15.254.62:43507)
0807 06:23:26.629195 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:26.629217 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:26.629261 (+ 44us) server_negotiation.cc:407] Connection header received
0807 06:23:26.630068 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:26.630096 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:26.630526 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:26.630906 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:26.631556 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:26.633047 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:26.633913 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:26.634595 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:26.634777 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:26.637832 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:26.637839 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:26.637855 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:23:26.637885 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:26.639994 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:26.640535 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:26.640543 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:26.640552 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:26.640731 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:26.641091 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:26.641094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:26.641096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:26.641610 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:26.641832 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:26.642214 (+ 382us) server_negotiation.cc:299] Negotiation successful
0807 06:23:26.642535 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":118,"threads_started":1}
I20260807 06:23:26.643831 31141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70e9b1eb672b496d962893879bff3b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e9b1eb672b496d962893879bff3b46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43507 } }
I20260807 06:23:26.643992 31141 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70e9b1eb672b496d962893879bff3b46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:23:26.644057 31141 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70e9b1eb672b496d962893879bff3b46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:23:26.644160 31141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70e9b1eb672b496d962893879bff3b46 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:23:26.645181 31141 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70e9b1eb672b496d962893879bff3b46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e9b1eb672b496d962893879bff3b46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43507 } }
I20260807 06:23:26.645565 31141 leader_election.cc:302] T 00000000000000000000000000000000 P 70e9b1eb672b496d962893879bff3b46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70e9b1eb672b496d962893879bff3b46; no voters:
I20260807 06:23:26.645970 31141 leader_election.cc:288] T 00000000000000000000000000000000 P 70e9b1eb672b496d962893879bff3b46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:23:26.646579 31147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70e9b1eb672b496d962893879bff3b46 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:23:26.646874 31147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70e9b1eb672b496d962893879bff3b46 [term 1 LEADER]: Becoming Leader. State: Replica: 70e9b1eb672b496d962893879bff3b46, State: Running, Role: LEADER
I20260807 06:23:26.647400 31141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70e9b1eb672b496d962893879bff3b46 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:23:26.647444 31147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70e9b1eb672b496d962893879bff3b46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e9b1eb672b496d962893879bff3b46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43507 } }
I20260807 06:23:26.649466 31147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e9b1eb672b496d962893879bff3b46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70e9b1eb672b496d962893879bff3b46. Latest consensus state: current_term: 1 leader_uuid: "70e9b1eb672b496d962893879bff3b46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e9b1eb672b496d962893879bff3b46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43507 } } }
I20260807 06:23:26.649798 31147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e9b1eb672b496d962893879bff3b46 [sys.catalog]: This master's current role is: LEADER
I20260807 06:23:26.650475 31149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e9b1eb672b496d962893879bff3b46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70e9b1eb672b496d962893879bff3b46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e9b1eb672b496d962893879bff3b46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43507 } } }
I20260807 06:23:26.650784 31149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e9b1eb672b496d962893879bff3b46 [sys.catalog]: This master's current role is: LEADER
W20260807 06:23:26.654508 31166 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 70e9b1eb672b496d962893879bff3b46: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:23:26.654592 31166 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:23:26.654671 31167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:23:26.655607 31167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:23:26.661621 31167 catalog_manager.cc:1347] Generated new cluster ID: 625c40a8c1c0412592f843203ef44b08
I20260807 06:23:26.661706 31167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:23:26.669359 31167 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:23:26.670689 31167 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:23:26.678198 31167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70e9b1eb672b496d962893879bff3b46: Generated new TSK 0
I20260807 06:23:26.679052 31167 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
I20260807 06:23:26.717209 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:0
--local_ip_for_outbound_sockets=127.15.254.1
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:43507
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:42951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:23:26.840282 31177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:23:26.840641 31177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:23:26.840734 31177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:23:26.845011 31177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:23:26.845072 31177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:23:26.845158 31177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:23:26.850734 31177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:42951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:43507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:23:26.851878 31177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:23:26.853048 31177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:23:26.863309 31182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:26.863315 31186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:26.863360 31184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:26.863673 31177 server_base.cc:1034] running on GCE node
I20260807 06:23:26.864156 31177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:23:26.865051 31177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:23:26.866257 31177 hybrid_clock.cc:648] HybridClock initialized: now 1786083806866215 us; error 57 us; skew 500 ppm
Aug 07 06:23:26 dist-test-slave-44d2 krb5kdc[31028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083806, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:23:26.869234 31177 init.cc:377] Logged in from keytab as kudu/127.15.254.1@KRBTEST.COM (short username kudu)
I20260807 06:23:26.870173 31177 webserver.cc:460] Webserver started at http://127.15.254.1:41433/ using document root <none> and password file <none>
I20260807 06:23:26.870767 31177 fs_manager.cc:359] Metadata directory not provided
I20260807 06:23:26.870831 31177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:23:26.871069 31177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:23:26.873328 31177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/data/instance:
uuid: "ab22ae0923a847d8957fb2dd0fb2e10c"
format_stamp: "Formatted at 2026-08-07 06:23:26 on dist-test-slave-44d2"
server_key: "3c70ad2cd3e5088cb806dea61b434c0e"
server_key_iv: "b4339eb4104c712ac2ea5ad071fc1432"
server_key_version: "encryptionkey@0"
I20260807 06:23:26.873922 31177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance:
uuid: "ab22ae0923a847d8957fb2dd0fb2e10c"
format_stamp: "Formatted at 2026-08-07 06:23:26 on dist-test-slave-44d2"
server_key: "3c70ad2cd3e5088cb806dea61b434c0e"
server_key_iv: "b4339eb4104c712ac2ea5ad071fc1432"
server_key_version: "encryptionkey@0"
I20260807 06:23:26.878113 31177 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:23:26.880833 31193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:26.882223 31177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:23:26.882354 31177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "ab22ae0923a847d8957fb2dd0fb2e10c"
format_stamp: "Formatted at 2026-08-07 06:23:26 on dist-test-slave-44d2"
server_key: "3c70ad2cd3e5088cb806dea61b434c0e"
server_key_iv: "b4339eb4104c712ac2ea5ad071fc1432"
server_key_version: "encryptionkey@0"
I20260807 06:23:26.882472 31177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:23:26.904177 31177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:23:26.907518 31177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:23:26.907739 31177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:23:26.908393 31177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:23:26.909312 31177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:23:26.909363 31177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:26.909435 31177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:23:26.909466 31177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:26.919296 31177 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:46837
I20260807 06:23:26.919317 31348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:46837 every 8 connection(s)
I20260807 06:23:26.920280 31177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:23:26.925616 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 31177
I20260807 06:23:26.925763 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance
I20260807 06:23:26.926115 16376 external_mini_cluster.cc:1428] Setting key 165a8706f9cf22a6922cf48c31696624
Aug 07 06:23:26 dist-test-slave-44d2 krb5kdc[31028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083806, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:23:26.937461 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:26.922383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:39091 (local address 127.15.254.62:43507)
0807 06:23:26.922546 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:26.922551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:26.923550 (+ 999us) server_negotiation.cc:407] Connection header received
0807 06:23:26.925097 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:26.925100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:26.925155 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:26.925285 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:26.927037 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:26.927814 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:26.929760 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:26.930016 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:26.930205 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:26.933090 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:26.933098 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:26.933104 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:23:26.933142 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:26.935021 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:26.935576 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:26.935580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:26.935584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:26.935685 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:26.936074 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:26.936079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:26.936081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:26.936345 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:26.936456 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:26.937081 (+ 625us) server_negotiation.cc:299] Negotiation successful
0807 06:23:26.937222 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
WARNING: no policy specified for kudu/127.15.254.2@KRBTEST.COM; defaulting to no policy
I20260807 06:23:26.938357 31351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:26.922877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:43507 (local address 127.15.254.1:39091)
0807 06:23:26.923382 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:26.923428 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:26.924866 (+ 1438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:26.925498 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:26.925509 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:26.926120 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:26.926834 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:26.926854 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:26.928033 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:26.928037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:26.929597 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:26.929614 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:26.930296 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:26.930302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:26.930495 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:26.931263 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:26.931281 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:26.932882 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:23:26.935150 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:23:26.935155 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:23:26.935170 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:23:26.935450 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:23:26.935792 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:23:26.935794 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:23:26.935796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:23:26.935960 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:23:26.936453 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:23:26.936458 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:23:26.936796 (+ 338us) client_negotiation.cc:769] Sending connection context
0807 06:23:26.936967 (+ 171us) client_negotiation.cc:240] Negotiation successful
0807 06:23:26.937258 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":119,"threads_started":1}
I20260807 06:23:26.940028 31349 heartbeater.cc:347] Connected to a master server at 127.15.254.62:43507
I20260807 06:23:26.940356 31349 heartbeater.cc:464] Registering TS with master...
I20260807 06:23:26.941051 31349 heartbeater.cc:511] Master 127.15.254.62:43507 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.2@KRBTEST.COM" created.
I20260807 06:23:26.943063 31089 ts_manager.cc:194] Registered new tserver with Master: ab22ae0923a847d8957fb2dd0fb2e10c (127.15.254.1:46837)
I20260807 06:23:26.944648 31089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.1@KRBTEST.COM'} at 127.15.254.1:39091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
I20260807 06:23:26.960301 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:0
--local_ip_for_outbound_sockets=127.15.254.2
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:43507
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:42951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:23:27.083936 31355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:23:27.084231 31355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:23:27.084275 31355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:23:27.088573 31355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:23:27.088636 31355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:23:27.088696 31355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:23:27.094007 31355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:42951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:43507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:23:27.095206 31355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:23:27.096257 31355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:23:27.106971 31364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:27.106971 31365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:27.107103 31355 server_base.cc:1034] running on GCE node
W20260807 06:23:27.107013 31367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:27.107653 31355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:23:27.108374 31355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:23:27.109557 31355 hybrid_clock.cc:648] HybridClock initialized: now 1786083807109537 us; error 27 us; skew 500 ppm
Aug 07 06:23:27 dist-test-slave-44d2 krb5kdc[31028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083807, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:23:27.112736 31355 init.cc:377] Logged in from keytab as kudu/127.15.254.2@KRBTEST.COM (short username kudu)
I20260807 06:23:27.113759 31355 webserver.cc:460] Webserver started at http://127.15.254.2:33117/ using document root <none> and password file <none>
I20260807 06:23:27.114370 31355 fs_manager.cc:359] Metadata directory not provided
I20260807 06:23:27.114442 31355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:23:27.114701 31355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:23:27.116954 31355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/data/instance:
uuid: "b9cd8606a8c64f87b7e1f04d9a4e1597"
format_stamp: "Formatted at 2026-08-07 06:23:27 on dist-test-slave-44d2"
server_key: "d89616f4b3cf450a579cd9279cf4b4b4"
server_key_iv: "0f3ba132af28be65648a8fad71c73b4b"
server_key_version: "encryptionkey@0"
I20260807 06:23:27.117569 31355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance:
uuid: "b9cd8606a8c64f87b7e1f04d9a4e1597"
format_stamp: "Formatted at 2026-08-07 06:23:27 on dist-test-slave-44d2"
server_key: "d89616f4b3cf450a579cd9279cf4b4b4"
server_key_iv: "0f3ba132af28be65648a8fad71c73b4b"
server_key_version: "encryptionkey@0"
I20260807 06:23:27.121701 31355 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260807 06:23:27.124630 31375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:27.126055 31355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 06:23:27.126206 31355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "b9cd8606a8c64f87b7e1f04d9a4e1597"
format_stamp: "Formatted at 2026-08-07 06:23:27 on dist-test-slave-44d2"
server_key: "d89616f4b3cf450a579cd9279cf4b4b4"
server_key_iv: "0f3ba132af28be65648a8fad71c73b4b"
server_key_version: "encryptionkey@0"
I20260807 06:23:27.126338 31355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:23:27.143992 31355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:23:27.147410 31355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:23:27.147694 31355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:23:27.148424 31355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:23:27.149612 31355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:23:27.149670 31355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:27.149750 31355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:23:27.149793 31355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:27.159230 31355 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:38619
I20260807 06:23:27.159260 31526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:38619 every 8 connection(s)
I20260807 06:23:27.160176 31355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:23:27.168794 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 31355
I20260807 06:23:27.168946 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance
I20260807 06:23:27.169359 16376 external_mini_cluster.cc:1428] Setting key f2bc3cde99e56f207db6f30db6de9e9e
Aug 07 06:23:27 dist-test-slave-44d2 krb5kdc[31028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083807, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:23:27.175937 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:27.162148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:60323 (local address 127.15.254.62:43507)
0807 06:23:27.162298 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:27.162302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:27.163221 (+ 919us) server_negotiation.cc:407] Connection header received
0807 06:23:27.164731 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:27.164734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:27.164798 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:27.164907 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:27.166171 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.166748 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.167970 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.168203 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.168320 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:27.171410 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:27.171417 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:27.171420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:23:27.171454 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:27.173297 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:27.174043 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:27.174046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:27.174047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:27.174131 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:27.174565 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:27.174568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:27.174569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:27.174888 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:27.174996 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:27.175692 (+ 696us) server_negotiation.cc:299] Negotiation successful
0807 06:23:27.175801 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 06:23:27.177326 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:27.162581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:43507 (local address 127.15.254.2:60323)
0807 06:23:27.163055 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:27.163110 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:27.164449 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:27.165107 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:27.165114 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:27.165498 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:27.166002 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:27.166015 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.166904 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.166907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:27.167836 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:27.167845 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.168369 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.168374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:27.168657 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:27.169482 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:27.169505 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:27.171196 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:23:27.173457 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:23:27.173466 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:23:27.173486 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:23:27.173912 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:23:27.174246 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:23:27.174251 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:23:27.174254 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:23:27.174464 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:23:27.175016 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:23:27.175025 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:23:27.175389 (+ 364us) client_negotiation.cc:769] Sending connection context
0807 06:23:27.175636 (+ 247us) client_negotiation.cc:240] Negotiation successful
0807 06:23:27.176036 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":123,"threads_started":1}
I20260807 06:23:27.178921 31527 heartbeater.cc:347] Connected to a master server at 127.15.254.62:43507
I20260807 06:23:27.179242 31527 heartbeater.cc:464] Registering TS with master...
I20260807 06:23:27.179960 31527 heartbeater.cc:511] Master 127.15.254.62:43507 requested a full tablet report, sending...
I20260807 06:23:27.181260 31089 ts_manager.cc:194] Registered new tserver with Master: b9cd8606a8c64f87b7e1f04d9a4e1597 (127.15.254.2:38619)
WARNING: no policy specified for kudu/127.15.254.3@KRBTEST.COM; defaulting to no policy
I20260807 06:23:27.182003 31089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.2@KRBTEST.COM'} at 127.15.254.2:60323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
I20260807 06:23:27.205050 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:0
--local_ip_for_outbound_sockets=127.15.254.3
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:43507
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:42951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:23:27.324060 31534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:23:27.324348 31534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:23:27.324396 31534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:23:27.328547 31534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:23:27.328610 31534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:23:27.328671 31534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:23:27.334550 31534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:42951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:43507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:23:27.335783 31534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:23:27.336952 31534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:23:27.347954 31541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:27.347972 31540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:23:27.348055 31543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:23:27.348521 31534 server_base.cc:1034] running on GCE node
I20260807 06:23:27.348973 31534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:23:27.349673 31534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:23:27.350875 31534 hybrid_clock.cc:648] HybridClock initialized: now 1786083807350842 us; error 49 us; skew 500 ppm
Aug 07 06:23:27 dist-test-slave-44d2 krb5kdc[31028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083807, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:23:27.354166 31534 init.cc:377] Logged in from keytab as kudu/127.15.254.3@KRBTEST.COM (short username kudu)
I20260807 06:23:27.355093 31534 webserver.cc:460] Webserver started at http://127.15.254.3:39269/ using document root <none> and password file <none>
I20260807 06:23:27.355638 31534 fs_manager.cc:359] Metadata directory not provided
I20260807 06:23:27.355698 31534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:23:27.355868 31534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:23:27.358042 31534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/data/instance:
uuid: "8af23703995e47c4bbc9a52bcb2a9945"
format_stamp: "Formatted at 2026-08-07 06:23:27 on dist-test-slave-44d2"
server_key: "50a5575c31312dee3d8a4bfe10091510"
server_key_iv: "1641276bc988c45c7d46d6e38be207f2"
server_key_version: "encryptionkey@0"
I20260807 06:23:27.358574 31534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance:
uuid: "8af23703995e47c4bbc9a52bcb2a9945"
format_stamp: "Formatted at 2026-08-07 06:23:27 on dist-test-slave-44d2"
server_key: "50a5575c31312dee3d8a4bfe10091510"
server_key_iv: "1641276bc988c45c7d46d6e38be207f2"
server_key_version: "encryptionkey@0"
I20260807 06:23:27.362448 31534 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:23:27.365077 31551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:27.366466 31534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:23:27.366573 31534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "8af23703995e47c4bbc9a52bcb2a9945"
format_stamp: "Formatted at 2026-08-07 06:23:27 on dist-test-slave-44d2"
server_key: "50a5575c31312dee3d8a4bfe10091510"
server_key_iv: "1641276bc988c45c7d46d6e38be207f2"
server_key_version: "encryptionkey@0"
I20260807 06:23:27.366665 31534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:23:27.385167 31534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:23:27.388876 31534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:23:27.389111 31534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:23:27.389746 31534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:23:27.390633 31534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:23:27.390704 31534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:27.390779 31534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:23:27.390830 31534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:23:27.400146 31534 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:37481
I20260807 06:23:27.400197 31707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:37481 every 8 connection(s)
I20260807 06:23:27.401275 31534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:23:27.403553 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 31534
I20260807 06:23:27.403692 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance
I20260807 06:23:27.404058 16376 external_mini_cluster.cc:1428] Setting key 7a8f7d761b1b07c417a061d43a233f3a
Aug 07 06:23:27 dist-test-slave-44d2 krb5kdc[31028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083807, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:23:27.415704 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:27.403288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:37571 (local address 127.15.254.62:43507)
0807 06:23:27.403427 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:27.403431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:27.404553 (+ 1122us) server_negotiation.cc:407] Connection header received
0807 06:23:27.406263 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:27.406269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:27.406327 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:27.406449 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:27.407700 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.408227 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.409417 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.409640 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.409723 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:27.412300 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:27.412304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:27.412307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:23:27.412354 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:27.413704 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:27.414197 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:27.414201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:27.414202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:27.414294 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:27.414635 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:27.414638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:27.414639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:27.414843 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:27.414934 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:27.415438 (+ 504us) server_negotiation.cc:299] Negotiation successful
0807 06:23:27.415551 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 06:23:27.416532 31710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:27.403761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:43507 (local address 127.15.254.3:37571)
0807 06:23:27.404322 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:27.404396 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:27.406050 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:27.406669 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:27.406676 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:27.407047 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:23:27.407551 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:27.407574 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.408337 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.408339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:27.409289 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:27.409299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.409777 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.409780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:27.409922 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:27.410539 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:23:27.410556 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:23:27.412136 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:23:27.413815 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:23:27.413819 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:23:27.413832 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:23:27.414082 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:23:27.414392 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:23:27.414394 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:23:27.414395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:23:27.414530 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:23:27.414942 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:23:27.414947 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:23:27.415168 (+ 221us) client_negotiation.cc:769] Sending connection context
0807 06:23:27.415324 (+ 156us) client_negotiation.cc:240] Negotiation successful
0807 06:23:27.415583 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":141,"threads_started":1}
I20260807 06:23:27.417860 31708 heartbeater.cc:347] Connected to a master server at 127.15.254.62:43507
I20260807 06:23:27.418161 31708 heartbeater.cc:464] Registering TS with master...
I20260807 06:23:27.418701 31708 heartbeater.cc:511] Master 127.15.254.62:43507 requested a full tablet report, sending...
I20260807 06:23:27.419721 31092 ts_manager.cc:194] Registered new tserver with Master: 8af23703995e47c4bbc9a52bcb2a9945 (127.15.254.3:37481)
I20260807 06:23:27.420320 31092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.3@KRBTEST.COM'} at 127.15.254.3:37571
I20260807 06:23:27.430717 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:23:27.440817 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:27.432832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55800 (local address 127.15.254.62:43507)
0807 06:23:27.432960 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:27.432963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:27.433239 (+ 276us) server_negotiation.cc:407] Connection header received
0807 06:23:27.433391 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:27.433397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:27.433444 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:27.433532 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:23:27.434279 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.434792 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.435673 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.435894 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.436010 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:27.437472 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:23:27.437475 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:23:27.437479 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:23:27.437512 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:23:27.439034 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:27.439476 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:27.439480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:27.439482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:27.439571 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:23:27.439869 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:23:27.439873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:23:27.439875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:23:27.440112 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:23:27.440261 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:27.440529 (+ 268us) server_negotiation.cc:299] Negotiation successful
0807 06:23:27.440645 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 06:23:27.444000 31092 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:23:27.447139 31092 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:23:27.462443 31722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:27.455992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:37481 (local address 127.0.0.1:41818)
0807 06:23:27.456413 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:27.456472 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:27.456645 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:27.458262 (+ 1617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:27.458265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:27.458284 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:23:27.458380 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:27.458386 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.459828 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.459832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:27.461023 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:27.461031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.461850 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.461853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:27.461963 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:27.462000 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 06:23:27.462135 (+ 135us) client_negotiation.cc:240] Negotiation successful
0807 06:23:27.462271 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":22,"spinlock_wait_cycles":7168,"thread_start_us":98,"threads_started":1}
I20260807 06:23:27.462447 31729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:27.457230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41818 (local address 127.15.254.3:37481)
0807 06:23:27.457806 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:27.457818 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:27.457840 (+ 22us) server_negotiation.cc:407] Connection header received
0807 06:23:27.457918 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:27.457925 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:27.458123 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:27.458277 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:23:27.458514 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.459665 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.461146 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.461714 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.461848 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:27.462021 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:27.462169 (+ 148us) server_negotiation.cc:299] Negotiation successful
0807 06:23:27.462271 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":178,"threads_started":1}
I20260807 06:23:27.462754 31725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:27.455992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39282 (local address 127.15.254.1:46837)
0807 06:23:27.456949 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:27.456958 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:27.456986 (+ 28us) server_negotiation.cc:407] Connection header received
0807 06:23:27.457089 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:27.457097 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:27.457315 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:27.457477 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:23:27.457968 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.459217 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.460982 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.461750 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.461864 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:27.462019 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:27.462472 (+ 453us) server_negotiation.cc:299] Negotiation successful
0807 06:23:27.462581 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":841,"thread_start_us":127,"threads_started":1}
I20260807 06:23:27.463145 31723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:27.455841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:46837 (local address 127.0.0.1:39282)
0807 06:23:27.456413 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:27.456472 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:27.456644 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:27.457646 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:27.457654 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:27.457681 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:23:27.457820 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:27.457830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.459369 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.459372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:27.460840 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:27.460853 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.462089 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.462093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:27.462286 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:27.462316 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 06:23:27.462907 (+ 591us) client_negotiation.cc:240] Negotiation successful
0807 06:23:27.462987 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":214,"spinlock_wait_cycles":4736,"thread_start_us":86,"threads_started":1}
I20260807 06:23:27.463804 31727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:27.457056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40286 (local address 127.15.254.2:38619)
0807 06:23:27.457519 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:27.457532 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:27.457557 (+ 25us) server_negotiation.cc:407] Connection header received
0807 06:23:27.457634 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:27.457639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:27.457783 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:27.457950 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:23:27.458171 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.459069 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.460836 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.461750 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.461864 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:27.462019 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:27.463578 (+ 1559us) server_negotiation.cc:299] Negotiation successful
0807 06:23:27.463682 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":128,"threads_started":1}
I20260807 06:23:27.464104 31726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:27.456613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:38619 (local address 127.0.0.1:40286)
0807 06:23:27.457411 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:27.457427 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:27.457521 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:27.457935 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:27.457938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:27.457958 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:23:27.458071 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:27.458080 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.459210 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.459213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:27.460694 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:27.460705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.462089 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.462093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:27.462281 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:27.462312 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 06:23:27.462389 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 06:23:27.462473 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":59,"thread_start_us":102,"threads_started":1}
I20260807 06:23:27.465835 31618 tablet_service.cc:1467] Processing CreateTablet for tablet a614198001ef4d9eb8c3e19370028b74 (DEFAULT_TABLE table=test-table [id=6926c0f0530b4838bd029543debf84af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:23:27.466348 31261 tablet_service.cc:1467] Processing CreateTablet for tablet a614198001ef4d9eb8c3e19370028b74 (DEFAULT_TABLE table=test-table [id=6926c0f0530b4838bd029543debf84af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:23:27.467034 31618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a614198001ef4d9eb8c3e19370028b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:23:27.466856 31433 tablet_service.cc:1467] Processing CreateTablet for tablet a614198001ef4d9eb8c3e19370028b74 (DEFAULT_TABLE table=test-table [id=6926c0f0530b4838bd029543debf84af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:23:27.467540 31261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a614198001ef4d9eb8c3e19370028b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:23:27.467928 31433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a614198001ef4d9eb8c3e19370028b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:23:27.474390 31730 tablet_bootstrap.cc:492] T a614198001ef4d9eb8c3e19370028b74 P 8af23703995e47c4bbc9a52bcb2a9945: Bootstrap starting.
I20260807 06:23:27.475975 31731 tablet_bootstrap.cc:492] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c: Bootstrap starting.
I20260807 06:23:27.476970 31734 tablet_bootstrap.cc:492] T a614198001ef4d9eb8c3e19370028b74 P b9cd8606a8c64f87b7e1f04d9a4e1597: Bootstrap starting.
I20260807 06:23:27.477008 31730 tablet_bootstrap.cc:654] T a614198001ef4d9eb8c3e19370028b74 P 8af23703995e47c4bbc9a52bcb2a9945: Neither blocks nor log segments found. Creating new log.
I20260807 06:23:27.478557 31731 tablet_bootstrap.cc:654] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c: Neither blocks nor log segments found. Creating new log.
I20260807 06:23:27.478717 31730 log.cc:824] T a614198001ef4d9eb8c3e19370028b74 P 8af23703995e47c4bbc9a52bcb2a9945: Log is configured to *not* fsync() on all Append() calls
I20260807 06:23:27.479480 31731 log.cc:824] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c: Log is configured to *not* fsync() on all Append() calls
I20260807 06:23:27.479643 31734 tablet_bootstrap.cc:654] T a614198001ef4d9eb8c3e19370028b74 P b9cd8606a8c64f87b7e1f04d9a4e1597: Neither blocks nor log segments found. Creating new log.
I20260807 06:23:27.480453 31734 log.cc:824] T a614198001ef4d9eb8c3e19370028b74 P b9cd8606a8c64f87b7e1f04d9a4e1597: Log is configured to *not* fsync() on all Append() calls
I20260807 06:23:27.481025 31730 tablet_bootstrap.cc:492] T a614198001ef4d9eb8c3e19370028b74 P 8af23703995e47c4bbc9a52bcb2a9945: No bootstrap required, opened a new log
I20260807 06:23:27.481266 31730 ts_tablet_manager.cc:1397] T a614198001ef4d9eb8c3e19370028b74 P 8af23703995e47c4bbc9a52bcb2a9945: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260807 06:23:27.481679 31731 tablet_bootstrap.cc:492] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c: No bootstrap required, opened a new log
I20260807 06:23:27.481876 31731 ts_tablet_manager.cc:1397] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 06:23:27.482527 31734 tablet_bootstrap.cc:492] T a614198001ef4d9eb8c3e19370028b74 P b9cd8606a8c64f87b7e1f04d9a4e1597: No bootstrap required, opened a new log
I20260807 06:23:27.482728 31734 ts_tablet_manager.cc:1397] T a614198001ef4d9eb8c3e19370028b74 P b9cd8606a8c64f87b7e1f04d9a4e1597: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 06:23:27.484637 31730 raft_consensus.cc:348] T a614198001ef4d9eb8c3e19370028b74 P 8af23703995e47c4bbc9a52bcb2a9945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab22ae0923a847d8957fb2dd0fb2e10c" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46837 } } peers { permanent_uuid: "8af23703995e47c4bbc9a52bcb2a9945" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 37481 } } peers { permanent_uuid: "b9cd8606a8c64f87b7e1f04d9a4e1597" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38619 } }
I20260807 06:23:27.484933 31730 raft_consensus.cc:374] T a614198001ef4d9eb8c3e19370028b74 P 8af23703995e47c4bbc9a52bcb2a9945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:23:27.484993 31730 raft_consensus.cc:729] T a614198001ef4d9eb8c3e19370028b74 P 8af23703995e47c4bbc9a52bcb2a9945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8af23703995e47c4bbc9a52bcb2a9945, State: Initialized, Role: FOLLOWER
I20260807 06:23:27.485518 31730 consensus_queue.cc:260] T a614198001ef4d9eb8c3e19370028b74 P 8af23703995e47c4bbc9a52bcb2a9945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab22ae0923a847d8957fb2dd0fb2e10c" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46837 } } peers { permanent_uuid: "8af23703995e47c4bbc9a52bcb2a9945" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 37481 } } peers { permanent_uuid: "b9cd8606a8c64f87b7e1f04d9a4e1597" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38619 } }
I20260807 06:23:27.486229 31708 heartbeater.cc:503] Master 127.15.254.62:43507 was elected leader, sending a full tablet report...
I20260807 06:23:27.486711 31730 ts_tablet_manager.cc:1428] T a614198001ef4d9eb8c3e19370028b74 P 8af23703995e47c4bbc9a52bcb2a9945: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260807 06:23:27.486627 31731 raft_consensus.cc:348] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab22ae0923a847d8957fb2dd0fb2e10c" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46837 } } peers { permanent_uuid: "8af23703995e47c4bbc9a52bcb2a9945" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 37481 } } peers { permanent_uuid: "b9cd8606a8c64f87b7e1f04d9a4e1597" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38619 } }
I20260807 06:23:27.486898 31731 raft_consensus.cc:374] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:23:27.486950 31731 raft_consensus.cc:729] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab22ae0923a847d8957fb2dd0fb2e10c, State: Initialized, Role: FOLLOWER
I20260807 06:23:27.486986 31734 raft_consensus.cc:348] T a614198001ef4d9eb8c3e19370028b74 P b9cd8606a8c64f87b7e1f04d9a4e1597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab22ae0923a847d8957fb2dd0fb2e10c" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46837 } } peers { permanent_uuid: "8af23703995e47c4bbc9a52bcb2a9945" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 37481 } } peers { permanent_uuid: "b9cd8606a8c64f87b7e1f04d9a4e1597" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38619 } }
I20260807 06:23:27.487211 31734 raft_consensus.cc:374] T a614198001ef4d9eb8c3e19370028b74 P b9cd8606a8c64f87b7e1f04d9a4e1597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:23:27.487272 31734 raft_consensus.cc:729] T a614198001ef4d9eb8c3e19370028b74 P b9cd8606a8c64f87b7e1f04d9a4e1597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9cd8606a8c64f87b7e1f04d9a4e1597, State: Initialized, Role: FOLLOWER
I20260807 06:23:27.487493 31731 consensus_queue.cc:260] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab22ae0923a847d8957fb2dd0fb2e10c" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46837 } } peers { permanent_uuid: "8af23703995e47c4bbc9a52bcb2a9945" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 37481 } } peers { permanent_uuid: "b9cd8606a8c64f87b7e1f04d9a4e1597" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38619 } }
I20260807 06:23:27.487735 31734 consensus_queue.cc:260] T a614198001ef4d9eb8c3e19370028b74 P b9cd8606a8c64f87b7e1f04d9a4e1597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab22ae0923a847d8957fb2dd0fb2e10c" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46837 } } peers { permanent_uuid: "8af23703995e47c4bbc9a52bcb2a9945" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 37481 } } peers { permanent_uuid: "b9cd8606a8c64f87b7e1f04d9a4e1597" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38619 } }
I20260807 06:23:27.488461 31527 heartbeater.cc:503] Master 127.15.254.62:43507 was elected leader, sending a full tablet report...
I20260807 06:23:27.488518 31731 ts_tablet_manager.cc:1428] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260807 06:23:27.488660 31734 ts_tablet_manager.cc:1428] T a614198001ef4d9eb8c3e19370028b74 P b9cd8606a8c64f87b7e1f04d9a4e1597: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260807 06:23:27.488840 31349 heartbeater.cc:503] Master 127.15.254.62:43507 was elected leader, sending a full tablet report...
I20260807 06:23:27.615100 31745 raft_consensus.cc:482] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:23:27.615392 31745 raft_consensus.cc:504] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab22ae0923a847d8957fb2dd0fb2e10c" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46837 } } peers { permanent_uuid: "8af23703995e47c4bbc9a52bcb2a9945" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 37481 } } peers { permanent_uuid: "b9cd8606a8c64f87b7e1f04d9a4e1597" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38619 } }
I20260807 06:23:27.616794 31745 leader_election.cc:288] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8af23703995e47c4bbc9a52bcb2a9945 (127.15.254.3:37481), b9cd8606a8c64f87b7e1f04d9a4e1597 (127.15.254.2:38619)
I20260807 06:23:27.621556 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:27.617133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:37481 (local address 127.15.254.1:56511)
0807 06:23:27.617492 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:27.617512 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:27.617662 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:27.617950 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:27.617953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:27.617979 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:23:27.618186 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:27.618194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.619097 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.619100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:27.620104 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:27.620114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.621092 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.621095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:27.621194 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:27.621218 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 06:23:27.621295 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 06:23:27.621375 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":44,"spinlock_wait_cycles":768,"thread_start_us":130,"threads_started":1}
I20260807 06:23:27.621572 31729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:27.617267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:56511 (local address 127.15.254.3:37481)
0807 06:23:27.617509 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:27.617516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:27.617625 (+ 109us) server_negotiation.cc:407] Connection header received
0807 06:23:27.617785 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:27.617791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:27.617852 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:27.617931 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:23:27.618338 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.618967 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.620280 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.620936 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.621042 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:27.621098 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:27.621341 (+ 243us) server_negotiation.cc:299] Negotiation successful
0807 06:23:27.621440 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260807 06:23:27.621646 31727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:27.617811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:41841 (local address 127.15.254.2:38619)
0807 06:23:27.617947 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:27.617951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:27.618047 (+ 96us) server_negotiation.cc:407] Connection header received
0807 06:23:27.618171 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:27.618174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:27.618232 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:27.618315 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:23:27.618587 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.619120 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.620249 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.620936 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.621051 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:27.621098 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:27.621451 (+ 353us) server_negotiation.cc:299] Negotiation successful
0807 06:23:27.621527 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 06:23:27.622059 31748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:27.617630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:38619 (local address 127.15.254.1:41841)
0807 06:23:27.617950 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:27.617964 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:23:27.618056 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:23:27.618340 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:23:27.618342 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:23:27.618361 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:23:27.618473 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:27.618479 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.619238 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.619241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:27.620097 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:23:27.620107 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.621170 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.621173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:23:27.621260 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:27.621315 (+ 55us) client_negotiation.cc:769] Sending connection context
0807 06:23:27.621820 (+ 505us) client_negotiation.cc:240] Negotiation successful
0807 06:23:27.621909 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":22,"thread_start_us":110,"threads_started":1}
I20260807 06:23:27.622352 31647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a614198001ef4d9eb8c3e19370028b74" candidate_uuid: "ab22ae0923a847d8957fb2dd0fb2e10c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8af23703995e47c4bbc9a52bcb2a9945" is_pre_election: true
I20260807 06:23:27.622694 31647 raft_consensus.cc:2454] T a614198001ef4d9eb8c3e19370028b74 P 8af23703995e47c4bbc9a52bcb2a9945 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab22ae0923a847d8957fb2dd0fb2e10c in term 0.
I20260807 06:23:27.623127 31466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a614198001ef4d9eb8c3e19370028b74" candidate_uuid: "ab22ae0923a847d8957fb2dd0fb2e10c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9cd8606a8c64f87b7e1f04d9a4e1597" is_pre_election: true
I20260807 06:23:27.623260 31196 leader_election.cc:302] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8af23703995e47c4bbc9a52bcb2a9945, ab22ae0923a847d8957fb2dd0fb2e10c; no voters:
I20260807 06:23:27.623440 31466 raft_consensus.cc:2454] T a614198001ef4d9eb8c3e19370028b74 P b9cd8606a8c64f87b7e1f04d9a4e1597 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab22ae0923a847d8957fb2dd0fb2e10c in term 0.
I20260807 06:23:27.623577 31745 raft_consensus.cc:2781] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:23:27.623669 31745 raft_consensus.cc:482] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:23:27.623709 31745 raft_consensus.cc:3037] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:23:27.625159 31745 raft_consensus.cc:504] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab22ae0923a847d8957fb2dd0fb2e10c" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46837 } } peers { permanent_uuid: "8af23703995e47c4bbc9a52bcb2a9945" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 37481 } } peers { permanent_uuid: "b9cd8606a8c64f87b7e1f04d9a4e1597" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38619 } }
I20260807 06:23:27.625514 31745 leader_election.cc:288] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c [CANDIDATE]: Term 1 election: Requested vote from peers 8af23703995e47c4bbc9a52bcb2a9945 (127.15.254.3:37481), b9cd8606a8c64f87b7e1f04d9a4e1597 (127.15.254.2:38619)
I20260807 06:23:27.626036 31466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a614198001ef4d9eb8c3e19370028b74" candidate_uuid: "ab22ae0923a847d8957fb2dd0fb2e10c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9cd8606a8c64f87b7e1f04d9a4e1597"
I20260807 06:23:27.626065 31647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a614198001ef4d9eb8c3e19370028b74" candidate_uuid: "ab22ae0923a847d8957fb2dd0fb2e10c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8af23703995e47c4bbc9a52bcb2a9945"
I20260807 06:23:27.626178 31647 raft_consensus.cc:3037] T a614198001ef4d9eb8c3e19370028b74 P 8af23703995e47c4bbc9a52bcb2a9945 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:23:27.626175 31466 raft_consensus.cc:3037] T a614198001ef4d9eb8c3e19370028b74 P b9cd8606a8c64f87b7e1f04d9a4e1597 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:23:27.627311 31466 raft_consensus.cc:2454] T a614198001ef4d9eb8c3e19370028b74 P b9cd8606a8c64f87b7e1f04d9a4e1597 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab22ae0923a847d8957fb2dd0fb2e10c in term 1.
I20260807 06:23:27.627580 31647 raft_consensus.cc:2454] T a614198001ef4d9eb8c3e19370028b74 P 8af23703995e47c4bbc9a52bcb2a9945 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab22ae0923a847d8957fb2dd0fb2e10c in term 1.
I20260807 06:23:27.627622 31196 leader_election.cc:302] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab22ae0923a847d8957fb2dd0fb2e10c, b9cd8606a8c64f87b7e1f04d9a4e1597; no voters:
I20260807 06:23:27.627811 31745 raft_consensus.cc:2781] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:23:27.628043 31745 raft_consensus.cc:686] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c [term 1 LEADER]: Becoming Leader. State: Replica: ab22ae0923a847d8957fb2dd0fb2e10c, State: Running, Role: LEADER
I20260807 06:23:27.628397 31745 consensus_queue.cc:237] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab22ae0923a847d8957fb2dd0fb2e10c" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46837 } } peers { permanent_uuid: "8af23703995e47c4bbc9a52bcb2a9945" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 37481 } } peers { permanent_uuid: "b9cd8606a8c64f87b7e1f04d9a4e1597" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38619 } }
I20260807 06:23:27.632269 31089 catalog_manager.cc:5495] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c reported cstate change: term changed from 0 to 1, leader changed from <none> to ab22ae0923a847d8957fb2dd0fb2e10c (127.15.254.1). New cstate: current_term: 1 leader_uuid: "ab22ae0923a847d8957fb2dd0fb2e10c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab22ae0923a847d8957fb2dd0fb2e10c" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8af23703995e47c4bbc9a52bcb2a9945" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 37481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9cd8606a8c64f87b7e1f04d9a4e1597" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38619 } health_report { overall_health: UNKNOWN } } }
I20260807 06:23:27.652143 31725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:27.648616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39292 (local address 127.15.254.1:46837)
0807 06:23:27.648804 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 06:23:27.648810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:23:27.648831 (+ 21us) server_negotiation.cc:407] Connection header received
0807 06:23:27.648958 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:23:27.648960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:23:27.649027 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:23:27.649131 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:23:27.649387 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.649907 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.650785 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:23:27.651009 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:23:27.651096 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:23:27.651402 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:23:27.651826 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:23:27.651943 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 06:23:27.651976 (+ 33us) server_negotiation.cc:299] Negotiation successful
0807 06:23:27.652031 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260807 06:23:27.652444 31709 tablet.cc:2369] T a614198001ef4d9eb8c3e19370028b74 P 8af23703995e47c4bbc9a52bcb2a9945: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:23:27.658516 31647 raft_consensus.cc:1261] T a614198001ef4d9eb8c3e19370028b74 P 8af23703995e47c4bbc9a52bcb2a9945 [term 1 FOLLOWER]: Refusing update from remote peer ab22ae0923a847d8957fb2dd0fb2e10c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:23:27.658548 31466 raft_consensus.cc:1261] T a614198001ef4d9eb8c3e19370028b74 P b9cd8606a8c64f87b7e1f04d9a4e1597 [term 1 FOLLOWER]: Refusing update from remote peer ab22ae0923a847d8957fb2dd0fb2e10c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:23:27.659386 31749 consensus_queue.cc:1035] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8af23703995e47c4bbc9a52bcb2a9945" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 37481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:23:27.659586 31745 consensus_queue.cc:1035] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9cd8606a8c64f87b7e1f04d9a4e1597" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 06:23:27.661469 31529 tablet.cc:2369] T a614198001ef4d9eb8c3e19370028b74 P b9cd8606a8c64f87b7e1f04d9a4e1597: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:23:27.668725 31755 mvcc.cc:205] Tried to move back new op lower bound from 7315799276161581056 to 7315799276049014784. Current Snapshot: MvccSnapshot[applied={T|T < 7315799276161581056}]
I20260807 06:23:27.669616 31757 mvcc.cc:205] Tried to move back new op lower bound from 7315799276161581056 to 7315799276049014784. Current Snapshot: MvccSnapshot[applied={T|T < 7315799276161581056}]
I20260807 06:23:27.676091 31088 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:23:27.676290 31088 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:23:27.678902 31088 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70e9b1eb672b496d962893879bff3b46: Sending DeleteTablet for 3 replicas of tablet a614198001ef4d9eb8c3e19370028b74
I20260807 06:23:27.679831 31261 tablet_service.cc:1514] Processing DeleteTablet for tablet a614198001ef4d9eb8c3e19370028b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:23:27 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:39282
I20260807 06:23:27.679821 31433 tablet_service.cc:1514] Processing DeleteTablet for tablet a614198001ef4d9eb8c3e19370028b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:23:27 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:40286
I20260807 06:23:27.679879 31618 tablet_service.cc:1514] Processing DeleteTablet for tablet a614198001ef4d9eb8c3e19370028b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:23:27 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:41818
I20260807 06:23:27.680379 31769 tablet_replica.cc:330] T a614198001ef4d9eb8c3e19370028b74 P ab22ae0923a847d8957fb2dd0fb2e10c: stopping tablet replica
I20260807 06:23:27.680542 31768 tablet_replica.cc:330] T a614198001ef4d9eb8c3e19370028b74 P b9cd8606a8c64f87b7e1f04d9a4e1597: stopping tablet replica
I20260807 06:23:27.681034 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 31177
I20260807 06:23:27.685508 31770 tablet_replica.cc:330] T a614198001ef4d9eb8c3e19370028b74 P 8af23703995e47c4bbc9a52bcb2a9945: stopping tablet replica
I20260807 06:23:27.685823 31770 raft_consensus.cc:2229] T a614198001ef4d9eb8c3e19370028b74 P 8af23703995e47c4bbc9a52bcb2a9945 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:23:27.686112 31770 raft_consensus.cc:2258] T a614198001ef4d9eb8c3e19370028b74 P 8af23703995e47c4bbc9a52bcb2a9945 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:23:27.680902 31768 raft_consensus.cc:2229] T a614198001ef4d9eb8c3e19370028b74 P b9cd8606a8c64f87b7e1f04d9a4e1597 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:23:27.687201 31768 raft_consensus.cc:2258] T a614198001ef4d9eb8c3e19370028b74 P b9cd8606a8c64f87b7e1f04d9a4e1597 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:23:27.687853 31770 ts_tablet_manager.cc:1905] T a614198001ef4d9eb8c3e19370028b74 P 8af23703995e47c4bbc9a52bcb2a9945: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:23:27.688654 31768 ts_tablet_manager.cc:1905] T a614198001ef4d9eb8c3e19370028b74 P b9cd8606a8c64f87b7e1f04d9a4e1597: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:23:27.690828 31770 ts_tablet_manager.cc:1918] T a614198001ef4d9eb8c3e19370028b74 P 8af23703995e47c4bbc9a52bcb2a9945: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:23:27.690936 31770 log.cc:1196] T a614198001ef4d9eb8c3e19370028b74 P 8af23703995e47c4bbc9a52bcb2a9945: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/a614198001ef4d9eb8c3e19370028b74
I20260807 06:23:27.690958 31768 ts_tablet_manager.cc:1918] T a614198001ef4d9eb8c3e19370028b74 P b9cd8606a8c64f87b7e1f04d9a4e1597: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:23:27.691059 31768 log.cc:1196] T a614198001ef4d9eb8c3e19370028b74 P b9cd8606a8c64f87b7e1f04d9a4e1597: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/a614198001ef4d9eb8c3e19370028b74
I20260807 06:23:27.691284 31770 ts_tablet_manager.cc:1939] T a614198001ef4d9eb8c3e19370028b74 P 8af23703995e47c4bbc9a52bcb2a9945: Deleting consensus metadata
I20260807 06:23:27.691411 31768 ts_tablet_manager.cc:1939] T a614198001ef4d9eb8c3e19370028b74 P b9cd8606a8c64f87b7e1f04d9a4e1597: Deleting consensus metadata
I20260807 06:23:27.692642 31071 catalog_manager.cc:4841] TS b9cd8606a8c64f87b7e1f04d9a4e1597 (127.15.254.2:38619): tablet a614198001ef4d9eb8c3e19370028b74 (table test-table [id=6926c0f0530b4838bd029543debf84af]) successfully deleted
I20260807 06:23:27.692955 31071 catalog_manager.cc:4841] TS 8af23703995e47c4bbc9a52bcb2a9945 (127.15.254.3:37481): tablet a614198001ef4d9eb8c3e19370028b74 (table test-table [id=6926c0f0530b4838bd029543debf84af]) successfully deleted
W20260807 06:23:27.694486 31074 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.254.1:46837 (error 108)
I20260807 06:23:27.695401 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 31355
I20260807 06:23:27.695703 31726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:23:27.695378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:46837 (local address 127.0.0.1:39298)
0807 06:23:27.695514 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0807 06:23:27.695594 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.1:46837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260807 06:23:27.695888 31074 catalog_manager.cc:4584] TS ab22ae0923a847d8957fb2dd0fb2e10c (127.15.254.1:46837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a614198001ef4d9eb8c3e19370028b74: Network error: Client connection negotiation failed: client connection to 127.15.254.1:46837: connect: Connection refused (error 111)
I20260807 06:23:27.705309 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 31534
I20260807 06:23:27.719511 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 31046
2026-08-07T06:23:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2663 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:23:27 dist-test-slave-44d2 krb5kdc[31774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:23:27 dist-test-slave-44d2 krb5kdc[31774](info): set up 2 sockets
Aug 07 06:23:27 dist-test-slave-44d2 krb5kdc[31774](info): commencing operation
krb5kdc: starting...
W20260807 06:23:28.964140 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.213s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:23:29 dist-test-slave-44d2 krb5kdc[31774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:23:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:23:29Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.254.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.254.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.254.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/rangeradmin_127.15.254.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.254.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/rangeradmin_127.15.254.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.254.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.254.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.254.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/rangerlookup_127.15.254.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.254.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/rangerlookup_127.15.254.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.254.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.254.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.254.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/HTTP_127.15.254.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.254.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/HTTP_127.15.254.20@KRBTEST.COM.keytab.
I20260807 06:23:29.159269 16376 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskZSVWny/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/postgres -l logfile start
2026-08-07 06:23:31.925 UTC [31816] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-07 06:23:31.925 UTC [31816] LOG: listening on IPv4 address "127.15.254.20", port 37327
2026-08-07 06:23:31.929 UTC [31816] LOG: listening on Unix socket "/tmp/.s.PGSQL.37327"
2026-08-07 06:23:31.954 UTC [31820] LOG: database system was shut down at 2026-08-07 06:23:30 UTC
2026-08-07 06:23:31.958 UTC [31816] LOG: database system is ready to accept connections
I20260807 06:23:33.136369 16376 mini_postgres.cc:96] Postgres bound to 37327
2026-08-07 06:23:33.144 UTC [31831] FATAL: database "slave" does not exist
127.15.254.20:37327 - accepting connections
I20260807 06:23:33.145530 16376 mini_ranger.cc:161] Starting Ranger...
I20260807 06:23:33.162524 16376 mini_ranger.cc:84] Created miniranger Postgres user
I20260807 06:23:33.720976 16376 mini_ranger.cc:87] Created ranger Postgres database
I20260807 06:23:33.721101 16376 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ranger-admin
2026-08-07 06:23:34,035 [I] DB FLAVOR :POSTGRES
2026-08-07 06:23:34,035 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 06:23:34,035 [I] Checking connection..
2026-08-07 06:23:34,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:37327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-07 06:23:34,321 [I] Checking connection passed.
2026-08-07 06:23:34,322 [I] --------- Verifying version history table ---------
2026-08-07 06:23:34,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:37327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 06:23:34,569 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 06:23:34,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:37327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 06:23:34,837 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 06:23:34,837 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-07 06:23:34,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:37327/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-07 06:23:35.087 UTC [31918] WARNING: there is no transaction in progress
2026-08-07 06:23:35,095 [I] create_dbversion_catalog.sql file imported successfully
2026-08-07 06:23:35,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:37327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 06:23:35,353 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-07 06:23:35,354 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 06:23:35,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:37327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-07 06:23:35,629 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:37327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-07 06:23:35,904 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:37327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-44d2.c.gcp-upstream.internal','N') ;"
2026-08-07 06:23:36,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:37327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 06:23:36,419 [I] Table x_portal_user does not exist in database ranger
2026-08-07 06:23:36,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:37327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 06:23:36,686 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-07 06:23:36,686 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:37327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 06:23:36,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:37327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 06:23:37,195 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-07 06:23:37,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:37327/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-07 06:23:38.300 UTC [32058] WARNING: there is no transaction in progress
2026-08-07 06:23:38.318 UTC [32058] WARNING: there is no transaction in progress
2026-08-07 06:23:38.337 UTC [32058] WARNING: there is no transaction in progress
2026-08-07 06:23:38.355 UTC [32058] WARNING: there is no transaction in progress
2026-08-07 06:23:38.376 UTC [32058] WARNING: there is no transaction in progress
2026-08-07 06:23:38.396 UTC [32058] WARNING: there is no transaction in progress
2026-08-07 06:23:38.458 UTC [32058] WARNING: there is no transaction in progress
2026-08-07 06:23:38.466 UTC [32058] WARNING: there is no transaction in progress
2026-08-07 06:23:38.484 UTC [32058] WARNING: there is no transaction in progress
2026-08-07 06:23:38.495 UTC [32058] WARNING: there is no transaction in progress
2026-08-07 06:23:38.507 UTC [32058] WARNING: there is no transaction in progress
2026-08-07 06:23:38.521 UTC [32058] WARNING: there is no transaction in progress
2026-08-07 06:23:38.531 UTC [32058] WARNING: there is no transaction in progress
2026-08-07 06:23:38.539 UTC [32058] WARNING: there is no transaction in progress
2026-08-07 06:23:38.547 UTC [32058] WARNING: there is no transaction in progress
2026-08-07 06:23:38.555 UTC [32058] WARNING: there is no transaction in progress
2026-08-07 06:23:39,312 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-07 06:23:39,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:37327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-07 06:23:39,573 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-07 06:23:39,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:37327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 06:23:39,828 [I] Table x_portal_user already exists in database 'ranger'
2026-08-07 06:23:39,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:37327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 06:23:40,136 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-07 06:23:40,137 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:37327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 06:23:40,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:37327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 06:23:40,683 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:37327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-44d2.c.gcp-upstream.internal';"
2026-08-07 06:23:40,966 [I] CORE_DB_SCHEMA import status has been updated
2026-08-07 06:23:40,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:37327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-07 06:23:41,256 [I] DB_PATCHES have already been applied
I20260807 06:23:41.263427 16376 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ranger-admin:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskZSVWny/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260807 06:23:41.263504 16376 mini_ranger.cc:193] Using host: 127.15.254.20
I20260807 06:23:41.268572 16376 mini_ranger.cc:227] Ranger admin URL: http://127.15.254.20:55457
Aug 07, 2026 6:23:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 6:23:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 6:23:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ranger-admin/ews/webapp
Aug 07, 2026 6:23:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 07, 2026 6:23:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 07, 2026 6:23:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ranger-admin/ews/webapp].
Aug 07, 2026 6:23:42 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260807 06:23:42.660089 16376 mini_ranger.cc:160] Time spent starting Ranger: real 9.515s user 0.001s sys 0.013s
Aug 07, 2026 6:24:03 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 07, 2026 6:24:03 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Aug 07, 2026 6:24:03 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 07, 2026 6:24:03 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 07, 2026 6:24:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 07, 2026 6:24:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 07, 2026 6:24:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 07, 2026 6:24:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 07, 2026 6:24:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 07, 2026 6:24:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 07, 2026 6:24:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 07, 2026 6:24:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 07, 2026 6:24:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 07, 2026 6:24:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 07, 2026 6:24:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 07, 2026 6:24:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 07, 2026 6:24:03 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 07, 2026 6:24:04 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260807 06:24:05.527824 16376 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.254.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.254.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.254.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/rangerkms_127.15.254.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.254.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/rangerkms_127.15.254.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.254.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.254.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.254.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/HTTP_127.15.254.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.254.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/HTTP_127.15.254.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 07 06:24:05 dist-test-slave-44d2 krb5kdc[31774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083845, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260807 06:24:05.626191 16376 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260807 06:24:05.638679 16376 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260807 06:24:06.669792 16376 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260807 06:24:06.669924 16376 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ranger-kms
I20260807 06:24:06.670049 16376 mini_ranger_kms.cc:227] Using postgres at 127.15.254.20:37327
2026-08-07 06:24:07,049 [I] DB FLAVOR :POSTGRES
2026-08-07 06:24:07,049 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 06:24:07,049 [I] Checking connection
2026-08-07 06:24:07,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:37327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-07 06:24:07,324 [I] connection success
2026-08-07 06:24:07,325 [I] --------- Verifying Ranger DB tables ---------
2026-08-07 06:24:07,325 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-07 06:24:07,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:37327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-07 06:24:07,584 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-07 06:24:07,584 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 06:24:07,584 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-07 06:24:07,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:37327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-07 06:24:07,892 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260807 06:24:08.442713 16376 mini_ranger_kms.cc:310] Created kms service
I20260807 06:24:08.691998 16376 mini_ranger_kms.cc:326] Created kudu user
I20260807 06:24:08.740172 16376 mini_ranger_kms.cc:343] Created rangerkms user
Aug 07 06:24:08 dist-test-slave-44d2 krb5kdc[31774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083845, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.254.20@KRBTEST.COM
I20260807 06:24:09.182489 16376 mini_ranger_kms.cc:384] Added ranger policy
I20260807 06:24:09.182657 16376 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ranger-kms:/tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260807 06:24:09.182678 16376 mini_ranger_kms.cc:240] Using host: 127.15.254.20
I20260807 06:24:09.188777 16376 mini_ranger_kms.cc:276] Ranger KMS PID: 32297
I20260807 06:24:09.188938 16376 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.254.20:43501
Aug 07, 2026 6:24:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 6:24:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 6:24:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 07, 2026 6:24:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 07, 2026 6:24:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ranger-kms/ews/webapp].
Aug 07, 2026 6:24:10 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 07 06:24:11 dist-test-slave-44d2 krb5kdc[31774](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786083851, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.254.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:24:11.884464 16376 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.258s user 0.011s sys 0.352s
I20260807 06:24:11.884644 16376 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260807 06:24:11.884770 16376 mini_ranger_kms.cc:401] 127.15.254.20:43501/kms/v1/keys
Aug 07 06:24:12 dist-test-slave-44d2 krb5kdc[31774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083851, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.254.20@KRBTEST.COM for HTTP/127.15.254.20@KRBTEST.COM
Aug 07 06:24:12 dist-test-slave-44d2 krb5kdc[31774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083851, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.254.20@KRBTEST.COM for HTTP/127.15.254.20@KRBTEST.COM
Aug 07 06:24:15 dist-test-slave-44d2 krb5kdc[31774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:24:15.120040 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:36211
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:34369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:36211
--ranger_config_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.254.20:43501/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:24:15.263427 32366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:24:15.263985 32366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:24:15.264093 32366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:24:15.273245 32366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:24:15.273350 32366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:24:15.273391 32366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:24:15.273424 32366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:24:15.273458 32366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:24:15.285125 32366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:34369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.254.20:43501/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:36211
--ranger_config_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:36211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:24:15.286733 32366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:24:15.288313 32366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:24:15.302824 32373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:24:15.302811 32374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:24:15.303146 32366 server_base.cc:1034] running on GCE node
W20260807 06:24:15.302811 32378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:24:15.303982 32366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:24:15.305333 32366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:24:15.306557 32366 hybrid_clock.cc:648] HybridClock initialized: now 1786083855306538 us; error 40 us; skew 500 ppm
Aug 07 06:24:15 dist-test-slave-44d2 krb5kdc[31774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083855, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:24:15.314342 32366 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:24:15.315340 32366 webserver.cc:460] Webserver started at http://127.15.254.62:40183/ using document root <none> and password file <none>
I20260807 06:24:15.315958 32366 fs_manager.cc:359] Metadata directory not provided
I20260807 06:24:15.316018 32366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:24:15.316256 32366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 06:24:15 dist-test-slave-44d2 krb5kdc[31774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083855, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for HTTP/127.15.254.20@KRBTEST.COM
I20260807 06:24:15.400918 32366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "80009b60900c4391a142bb30893f9a46"
format_stamp: "Formatted at 2026-08-07 06:24:15 on dist-test-slave-44d2"
server_key: "701997beaa80d098f395ef285f94835a"
server_key_iv: "e68fb44c7fc7bec7106305e830eac7ab"
server_key_version: "kuduclusterkey@0"
I20260807 06:24:15.401844 32366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "80009b60900c4391a142bb30893f9a46"
format_stamp: "Formatted at 2026-08-07 06:24:15 on dist-test-slave-44d2"
server_key: "701997beaa80d098f395ef285f94835a"
server_key_iv: "e68fb44c7fc7bec7106305e830eac7ab"
server_key_version: "kuduclusterkey@0"
I20260807 06:24:15.407586 32366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260807 06:24:15.432282 32388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:24:15.434365 32366 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260807 06:24:15.434554 32366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "80009b60900c4391a142bb30893f9a46"
format_stamp: "Formatted at 2026-08-07 06:24:15 on dist-test-slave-44d2"
server_key: "701997beaa80d098f395ef285f94835a"
server_key_iv: "e68fb44c7fc7bec7106305e830eac7ab"
server_key_version: "kuduclusterkey@0"
I20260807 06:24:15.434728 32366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:24:15.464046 32366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:24:15.467517 32366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:24:15.467726 32366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:24:15.477206 32366 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:36211
I20260807 06:24:15.477244 32452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:36211 every 8 connection(s)
I20260807 06:24:15.478266 32366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:24:15.481277 32453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:24:15.482200 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 32366
I20260807 06:24:15.482352 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:24:15.482686 16376 external_mini_cluster.cc:1428] Setting key 5a33bd9480aafab2d9bfc50275bea970
I20260807 06:24:15.488142 32453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80009b60900c4391a142bb30893f9a46: Bootstrap starting.
I20260807 06:24:15.490617 32453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80009b60900c4391a142bb30893f9a46: Neither blocks nor log segments found. Creating new log.
I20260807 06:24:15.491884 32453 log.cc:824] T 00000000000000000000000000000000 P 80009b60900c4391a142bb30893f9a46: Log is configured to *not* fsync() on all Append() calls
Aug 07 06:24:15 dist-test-slave-44d2 krb5kdc[31774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:24:15.494465 32453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80009b60900c4391a142bb30893f9a46: No bootstrap required, opened a new log
I20260807 06:24:15.498737 32453 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80009b60900c4391a142bb30893f9a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80009b60900c4391a142bb30893f9a46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36211 } }
I20260807 06:24:15.498993 32453 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80009b60900c4391a142bb30893f9a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:24:15.499068 32453 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80009b60900c4391a142bb30893f9a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80009b60900c4391a142bb30893f9a46, State: Initialized, Role: FOLLOWER
I20260807 06:24:15.499131 32458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:24:15.484283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40638 (local address 127.15.254.62:36211)
0807 06:24:15.484975 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0807 06:24:15.484990 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 06:24:15.485031 (+ 41us) server_negotiation.cc:407] Connection header received
0807 06:24:15.485820 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:24:15.485839 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:24:15.486179 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:24:15.486535 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:24:15.487408 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:24:15.488607 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:24:15.489474 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:24:15.489956 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:24:15.490154 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:24:15.493474 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:24:15.493480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:24:15.493494 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:24:15.493520 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:24:15.495927 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:24:15.496418 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:24:15.496423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:24:15.496428 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:24:15.496601 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:24:15.496971 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:24:15.496975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:24:15.496976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:24:15.497414 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:24:15.497605 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0807 06:24:15.497981 (+ 376us) server_negotiation.cc:299] Negotiation successful
0807 06:24:15.498330 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":353,"threads_started":1}
I20260807 06:24:15.499590 32453 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80009b60900c4391a142bb30893f9a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80009b60900c4391a142bb30893f9a46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36211 } }
I20260807 06:24:15.499755 32453 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80009b60900c4391a142bb30893f9a46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:24:15.499819 32453 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80009b60900c4391a142bb30893f9a46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:24:15.499912 32453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80009b60900c4391a142bb30893f9a46 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:24:15.500975 32453 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80009b60900c4391a142bb30893f9a46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80009b60900c4391a142bb30893f9a46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36211 } }
I20260807 06:24:15.501328 32453 leader_election.cc:302] T 00000000000000000000000000000000 P 80009b60900c4391a142bb30893f9a46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80009b60900c4391a142bb30893f9a46; no voters:
I20260807 06:24:15.501726 32453 leader_election.cc:288] T 00000000000000000000000000000000 P 80009b60900c4391a142bb30893f9a46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:24:15.501832 32461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80009b60900c4391a142bb30893f9a46 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:24:15.502103 32461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80009b60900c4391a142bb30893f9a46 [term 1 LEADER]: Becoming Leader. State: Replica: 80009b60900c4391a142bb30893f9a46, State: Running, Role: LEADER
I20260807 06:24:15.502449 32461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80009b60900c4391a142bb30893f9a46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80009b60900c4391a142bb30893f9a46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36211 } }
I20260807 06:24:15.502964 32453 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80009b60900c4391a142bb30893f9a46 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:24:15.504181 32463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80009b60900c4391a142bb30893f9a46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80009b60900c4391a142bb30893f9a46. Latest consensus state: current_term: 1 leader_uuid: "80009b60900c4391a142bb30893f9a46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80009b60900c4391a142bb30893f9a46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36211 } } }
I20260807 06:24:15.504311 32463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80009b60900c4391a142bb30893f9a46 [sys.catalog]: This master's current role is: LEADER
I20260807 06:24:15.504698 32462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80009b60900c4391a142bb30893f9a46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80009b60900c4391a142bb30893f9a46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80009b60900c4391a142bb30893f9a46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36211 } } }
I20260807 06:24:15.504866 32462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80009b60900c4391a142bb30893f9a46 [sys.catalog]: This master's current role is: LEADER
I20260807 06:24:15.504752 32470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:24:15.508159 32470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:24:15.508790 32453 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 06:24:15.518986 32470 catalog_manager.cc:1347] Generated new cluster ID: 5e479c031fc2495ab7d3e9ce56f365c6
I20260807 06:24:15.519085 32470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:24:15.539878 32470 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:24:15.541347 32470 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:24:15.554121 32470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80009b60900c4391a142bb30893f9a46: Generated new TSK 0
I20260807 06:24:15.555029 32470 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
I20260807 06:24:16.336071 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.254.20:43501/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:0
--local_ip_for_outbound_sockets=127.15.254.1
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:36211
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:34369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:24:16.500638 32513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:24:16.500967 32513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:24:16.501104 32513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:24:16.506021 32513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:24:16.506121 32513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:24:16.506217 32513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:24:16.513489 32513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:34369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.254.20:43501/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:36211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:24:16.514783 32513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:24:16.516032 32513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:24:16.527861 32523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:24:16.528090 32521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:24:16.527849 32525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:24:16.529184 32513 server_base.cc:1034] running on GCE node
I20260807 06:24:16.529663 32513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:24:16.530508 32513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:24:16.531738 32513 hybrid_clock.cc:648] HybridClock initialized: now 1786083856531708 us; error 65 us; skew 500 ppm
Aug 07 06:24:16 dist-test-slave-44d2 krb5kdc[31774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083856, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:24:16.534847 32513 init.cc:377] Logged in from keytab as kudu/127.15.254.1@KRBTEST.COM (short username kudu)
I20260807 06:24:16.535894 32513 webserver.cc:460] Webserver started at http://127.15.254.1:35669/ using document root <none> and password file <none>
I20260807 06:24:16.536547 32513 fs_manager.cc:359] Metadata directory not provided
I20260807 06:24:16.536625 32513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:24:16.536851 32513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 06:24:16 dist-test-slave-44d2 krb5kdc[31774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083856, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for HTTP/127.15.254.20@KRBTEST.COM
I20260807 06:24:16.554072 32513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-0/data/instance:
uuid: "1f4baaef44154f3a8728c36555a75a0d"
format_stamp: "Formatted at 2026-08-07 06:24:16 on dist-test-slave-44d2"
server_key: "880ee42d1eaaec7afc4c5f5b1d10d1b9"
server_key_iv: "f89fef2988ca21d5784c4c2e3e512f9d"
server_key_version: "kuduclusterkey@0"
I20260807 06:24:16.557888 32513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance:
uuid: "1f4baaef44154f3a8728c36555a75a0d"
format_stamp: "Formatted at 2026-08-07 06:24:16 on dist-test-slave-44d2"
server_key: "880ee42d1eaaec7afc4c5f5b1d10d1b9"
server_key_iv: "f89fef2988ca21d5784c4c2e3e512f9d"
server_key_version: "kuduclusterkey@0"
I20260807 06:24:16.570291 32513 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.004s sys 0.000s
I20260807 06:24:16.594729 32532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:24:16.596657 32513 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260807 06:24:16.596807 32513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "1f4baaef44154f3a8728c36555a75a0d"
format_stamp: "Formatted at 2026-08-07 06:24:16 on dist-test-slave-44d2"
server_key: "880ee42d1eaaec7afc4c5f5b1d10d1b9"
server_key_iv: "f89fef2988ca21d5784c4c2e3e512f9d"
server_key_version: "kuduclusterkey@0"
I20260807 06:24:16.596963 32513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:24:16.625310 32513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:24:16.629505 32513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:24:16.629792 32513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:24:16.630774 32513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:24:16.632005 32513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:24:16.632117 32513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:24:16.632261 32513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:24:16.632391 32513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:24:16.648452 32513 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:38203
I20260807 06:24:16.649995 32513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:24:16.652016 32695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:38203 every 8 connection(s)
I20260807 06:24:16.653476 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 32513
I20260807 06:24:16.653613 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance
I20260807 06:24:16.653955 16376 external_mini_cluster.cc:1428] Setting key a224ce073480c650d6667571373afb93
Aug 07 06:24:16 dist-test-slave-44d2 krb5kdc[31774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083856, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.254.2@KRBTEST.COM; defaulting to no policy
I20260807 06:24:16.672091 32698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:24:16.654966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:50737 (local address 127.15.254.62:36211)
0807 06:24:16.655279 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0807 06:24:16.655284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:24:16.656182 (+ 898us) server_negotiation.cc:407] Connection header received
0807 06:24:16.657816 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:24:16.657820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:24:16.657875 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:24:16.657939 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:24:16.659849 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:24:16.660830 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:24:16.662735 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:24:16.663173 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:24:16.663290 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:24:16.667123 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:24:16.667129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:24:16.667133 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:24:16.667171 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:24:16.669425 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:24:16.670197 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:24:16.670201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:24:16.670203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:24:16.670299 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:24:16.670738 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:24:16.670742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:24:16.670743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:24:16.671026 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:24:16.671159 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 06:24:16.671728 (+ 569us) server_negotiation.cc:299] Negotiation successful
0807 06:24:16.671871 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":116,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.2@KRBTEST.COM" created.
I20260807 06:24:16.673506 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:24:16.655485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:36211 (local address 127.15.254.1:50737)
0807 06:24:16.656023 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0807 06:24:16.656079 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 06:24:16.657256 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:24:16.658404 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:24:16.658415 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:24:16.658903 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:24:16.659665 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:24:16.659689 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:24:16.661005 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:24:16.661010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:24:16.662582 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:24:16.662598 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:24:16.663567 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:24:16.663572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:24:16.663809 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:24:16.664629 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:24:16.664655 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:24:16.666881 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:24:16.669610 (+ 2729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:24:16.669617 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:24:16.669639 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:24:16.670049 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:24:16.670442 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:24:16.670446 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:24:16.670448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:24:16.670616 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:24:16.671157 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:24:16.671165 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:24:16.671461 (+ 296us) client_negotiation.cc:769] Sending connection context
0807 06:24:16.671684 (+ 223us) client_negotiation.cc:240] Negotiation successful
0807 06:24:16.672047 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":151,"threads_started":1}
I20260807 06:24:16.677249 32696 heartbeater.cc:347] Connected to a master server at 127.15.254.62:36211
I20260807 06:24:16.677537 32696 heartbeater.cc:464] Registering TS with master...
I20260807 06:24:16.678189 32696 heartbeater.cc:511] Master 127.15.254.62:36211 requested a full tablet report, sending...
I20260807 06:24:16.680289 32409 ts_manager.cc:194] Registered new tserver with Master: 1f4baaef44154f3a8728c36555a75a0d (127.15.254.1:38203)
I20260807 06:24:16.682128 32409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.1@KRBTEST.COM'} at 127.15.254.1:50737
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
I20260807 06:24:16.698015 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.254.20:43501/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:0
--local_ip_for_outbound_sockets=127.15.254.2
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:36211
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:34369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 06:24:16 dist-test-slave-44d2 krb5kdc[31774](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786083856, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260807 06:24:16.846473 32704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:24:16.846796 32704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:24:16.846889 32704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:24:16.851440 32704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:24:16.851527 32704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:24:16.851643 32704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:24:16.857270 32704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:34369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.254.20:43501/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:36211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:24:16.858570 32704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:24:16.859685 32704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:24:16.871701 32714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:24:16.871695 32713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:24:16.871695 32717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:24:16.872314 32704 server_base.cc:1034] running on GCE node
I20260807 06:24:16.872814 32704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:24:16.873629 32704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:24:16.874797 32704 hybrid_clock.cc:648] HybridClock initialized: now 1786083856874761 us; error 54 us; skew 500 ppm
Aug 07 06:24:16 dist-test-slave-44d2 krb5kdc[31774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083856, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:24:16.878605 32704 init.cc:377] Logged in from keytab as kudu/127.15.254.2@KRBTEST.COM (short username kudu)
I20260807 06:24:16.879730 32704 webserver.cc:460] Webserver started at http://127.15.254.2:38261/ using document root <none> and password file <none>
I20260807 06:24:16.880551 32704 fs_manager.cc:359] Metadata directory not provided
I20260807 06:24:16.880645 32704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:24:16.880903 32704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 06:24:16 dist-test-slave-44d2 krb5kdc[31774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083856, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for HTTP/127.15.254.20@KRBTEST.COM
I20260807 06:24:16.896217 32704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-1/data/instance:
uuid: "833a4325a467456e92ee4de0bf47a196"
format_stamp: "Formatted at 2026-08-07 06:24:16 on dist-test-slave-44d2"
server_key: "f92741275e96cb17d3241e0b59aa372e"
server_key_iv: "f67d51f160aaafd924d2355ed74fb9fc"
server_key_version: "kuduclusterkey@0"
I20260807 06:24:16.897070 32704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance:
uuid: "833a4325a467456e92ee4de0bf47a196"
format_stamp: "Formatted at 2026-08-07 06:24:16 on dist-test-slave-44d2"
server_key: "f92741275e96cb17d3241e0b59aa372e"
server_key_iv: "f67d51f160aaafd924d2355ed74fb9fc"
server_key_version: "kuduclusterkey@0"
I20260807 06:24:16.906104 32704 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.001s sys 0.003s
I20260807 06:24:16.941104 32726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:24:16.942987 32704 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.001s
I20260807 06:24:16.943162 32704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "833a4325a467456e92ee4de0bf47a196"
format_stamp: "Formatted at 2026-08-07 06:24:16 on dist-test-slave-44d2"
server_key: "f92741275e96cb17d3241e0b59aa372e"
server_key_iv: "f67d51f160aaafd924d2355ed74fb9fc"
server_key_version: "kuduclusterkey@0"
I20260807 06:24:16.943329 32704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:24:16.977993 32704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:24:16.982177 32704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:24:16.982388 32704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:24:16.983183 32704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:24:16.984279 32704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:24:16.984344 32704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:24:16.984409 32704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:24:16.984440 32704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:24:17.001596 32704 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:36381
I20260807 06:24:17.003579 32704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:24:17.009630 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 32704
I20260807 06:24:17.009766 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance
I20260807 06:24:17.010140 16376 external_mini_cluster.cc:1428] Setting key d30d6b0d74bce13df90e342173801d04
I20260807 06:24:17.005581 410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:36381 every 8 connection(s)
Aug 07 06:24:17 dist-test-slave-44d2 krb5kdc[31774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083856, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:24:17.035128 32698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:24:17.010650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:41967 (local address 127.15.254.62:36211)
0807 06:24:17.011152 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0807 06:24:17.011156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:24:17.015271 (+ 4115us) server_negotiation.cc:407] Connection header received
0807 06:24:17.016778 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:24:17.016782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:24:17.016852 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:24:17.016908 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:24:17.019008 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.019935 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.021774 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.022153 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.022240 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:24:17.028664 (+ 6424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:24:17.028669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:24:17.028672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:24:17.028705 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:24:17.030700 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:24:17.031452 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:24:17.031456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:24:17.031458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:24:17.031552 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:24:17.032032 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:24:17.032035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:24:17.032037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:24:17.032290 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:24:17.032375 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 06:24:17.034770 (+ 2395us) server_negotiation.cc:299] Negotiation successful
0807 06:24:17.034942 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386}
I20260807 06:24:17.037215 414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:24:17.008108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:36211 (local address 127.15.254.2:41967)
0807 06:24:17.015093 (+ 6985us) negotiation.cc:107] Waiting for socket to connect
0807 06:24:17.015161 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0807 06:24:17.016455 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:24:17.017403 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:24:17.017413 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:24:17.017833 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:24:17.018670 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:24:17.018694 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.020114 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.020118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:24:17.021621 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:24:17.021634 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.022507 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.022511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:24:17.022722 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:24:17.024655 (+ 1933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:24:17.024680 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:24:17.026745 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:24:17.030905 (+ 4160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:24:17.030913 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:24:17.030934 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:24:17.031320 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:24:17.031681 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:24:17.031684 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:24:17.031686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:24:17.031899 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:24:17.032450 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:24:17.032458 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:24:17.032897 (+ 439us) client_negotiation.cc:769] Sending connection context
0807 06:24:17.035641 (+ 2744us) client_negotiation.cc:240] Negotiation successful
0807 06:24:17.036040 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3593,"thread_start_us":172,"threads_started":1}
I20260807 06:24:17.038769 412 heartbeater.cc:347] Connected to a master server at 127.15.254.62:36211
I20260807 06:24:17.039041 412 heartbeater.cc:464] Registering TS with master...
I20260807 06:24:17.039741 412 heartbeater.cc:511] Master 127.15.254.62:36211 requested a full tablet report, sending...
I20260807 06:24:17.041145 32409 ts_manager.cc:194] Registered new tserver with Master: 833a4325a467456e92ee4de0bf47a196 (127.15.254.2:36381)
I20260807 06:24:17.041931 32409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.2@KRBTEST.COM'} at 127.15.254.2:41967
WARNING: no policy specified for kudu/127.15.254.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
I20260807 06:24:17.094478 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.254.20:43501/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:0
--local_ip_for_outbound_sockets=127.15.254.3
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:36211
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:34369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:24:17.283833 421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:24:17.284731 421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:24:17.284898 421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:24:17.291419 421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:24:17.291640 421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:24:17.291792 421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:24:17.300129 421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:34369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.254.20:43501/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:36211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:24:17.302006 421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:24:17.303644 421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 06:24:17.318876 421 server_base.cc:1034] running on GCE node
W20260807 06:24:17.318929 431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:24:17.320999 428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:24:17.325019 426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:24:17.325677 421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:24:17.326599 421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:24:17.333936 421 hybrid_clock.cc:648] HybridClock initialized: now 1786083857332768 us; error 1166 us; skew 500 ppm
Aug 07 06:24:17 dist-test-slave-44d2 krb5kdc[31774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083857, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:24:17.337788 421 init.cc:377] Logged in from keytab as kudu/127.15.254.3@KRBTEST.COM (short username kudu)
I20260807 06:24:17.338994 421 webserver.cc:460] Webserver started at http://127.15.254.3:42759/ using document root <none> and password file <none>
I20260807 06:24:17.339778 421 fs_manager.cc:359] Metadata directory not provided
I20260807 06:24:17.339936 421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:24:17.340217 421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 06:24:17 dist-test-slave-44d2 krb5kdc[31774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083857, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for HTTP/127.15.254.20@KRBTEST.COM
Aug 07 06:24:17 dist-test-slave-44d2 krb5kdc[31774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083856, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for HTTP/127.15.254.20@KRBTEST.COM
I20260807 06:24:17.369586 421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-2/data/instance:
uuid: "3cd0216079174a878b8d73097e522998"
format_stamp: "Formatted at 2026-08-07 06:24:17 on dist-test-slave-44d2"
server_key: "03c76d243482830414b3a8229a779cfb"
server_key_iv: "a765416408af5f525ec6010540cef275"
server_key_version: "kuduclusterkey@0"
I20260807 06:24:17.378026 421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance:
uuid: "3cd0216079174a878b8d73097e522998"
format_stamp: "Formatted at 2026-08-07 06:24:17 on dist-test-slave-44d2"
server_key: "03c76d243482830414b3a8229a779cfb"
server_key_iv: "a765416408af5f525ec6010540cef275"
server_key_version: "kuduclusterkey@0"
I20260807 06:24:17.388963 421 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.001s sys 0.004s
I20260807 06:24:17.410202 438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:24:17.411883 421 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260807 06:24:17.412086 421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "3cd0216079174a878b8d73097e522998"
format_stamp: "Formatted at 2026-08-07 06:24:17 on dist-test-slave-44d2"
server_key: "03c76d243482830414b3a8229a779cfb"
server_key_iv: "a765416408af5f525ec6010540cef275"
server_key_version: "kuduclusterkey@0"
I20260807 06:24:17.412286 421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:24:17.440725 421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:24:17.452700 421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:24:17.453037 421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:24:17.453773 421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:24:17.454908 421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:24:17.455016 421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:24:17.455112 421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:24:17.455175 421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:24:17.480727 421 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:43685
I20260807 06:24:17.482118 421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:24:17.483201 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 421
I20260807 06:24:17.483350 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance
I20260807 06:24:17.483705 16376 external_mini_cluster.cc:1428] Setting key 29ed470e1ea8a92e3e998208b05db6d1
I20260807 06:24:17.486722 599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:43685 every 8 connection(s)
Aug 07 06:24:17 dist-test-slave-44d2 krb5kdc[31774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083857, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:24:17.508100 32698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:24:17.490396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:48139 (local address 127.15.254.62:36211)
0807 06:24:17.490514 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0807 06:24:17.490519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:24:17.490535 (+ 16us) server_negotiation.cc:407] Connection header received
0807 06:24:17.491209 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:24:17.491213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:24:17.491265 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:24:17.491314 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:24:17.493218 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.494020 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.495655 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.495971 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.496046 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:24:17.503220 (+ 7174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:24:17.503225 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:24:17.503228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:24:17.503259 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:24:17.505427 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:24:17.506175 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:24:17.506179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:24:17.506181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:24:17.506282 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:24:17.506717 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:24:17.506720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:24:17.506722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:24:17.506976 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:24:17.507061 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 06:24:17.507787 (+ 726us) server_negotiation.cc:299] Negotiation successful
0807 06:24:17.507935 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 06:24:17.509877 603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:24:17.489154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:36211 (local address 127.15.254.3:48139)
0807 06:24:17.489838 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0807 06:24:17.489901 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0807 06:24:17.490956 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:24:17.491749 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:24:17.491758 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:24:17.492123 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:24:17.493047 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:24:17.493070 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.494182 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.494185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:24:17.495510 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:24:17.495521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.496257 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.496261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:24:17.496544 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:24:17.500666 (+ 4122us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:24:17.500689 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:24:17.502990 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:24:17.505624 (+ 2634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:24:17.505631 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:24:17.505651 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:24:17.506048 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:24:17.506407 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:24:17.506410 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:24:17.506412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:24:17.506604 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:24:17.507132 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:24:17.507139 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:24:17.507484 (+ 345us) client_negotiation.cc:769] Sending connection context
0807 06:24:17.508390 (+ 906us) client_negotiation.cc:240] Negotiation successful
0807 06:24:17.508782 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":198,"threads_started":1}
I20260807 06:24:17.513484 600 heartbeater.cc:347] Connected to a master server at 127.15.254.62:36211
I20260807 06:24:17.513760 600 heartbeater.cc:464] Registering TS with master...
I20260807 06:24:17.514397 600 heartbeater.cc:511] Master 127.15.254.62:36211 requested a full tablet report, sending...
I20260807 06:24:17.515647 32409 ts_manager.cc:194] Registered new tserver with Master: 3cd0216079174a878b8d73097e522998 (127.15.254.3:43685)
I20260807 06:24:17.516501 32409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.3@KRBTEST.COM'} at 127.15.254.3:48139
I20260807 06:24:17.521571 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:24:17.536711 32698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:24:17.526106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40650 (local address 127.15.254.62:36211)
0807 06:24:17.526231 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0807 06:24:17.526236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:24:17.526902 (+ 666us) server_negotiation.cc:407] Connection header received
0807 06:24:17.527096 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:24:17.527099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:24:17.527151 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:24:17.527245 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:24:17.528303 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.529158 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.530418 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.530749 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.530826 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:24:17.532834 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:24:17.532838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:24:17.532840 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:24:17.532870 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:24:17.534714 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:24:17.535215 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:24:17.535219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:24:17.535221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:24:17.535308 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:24:17.535620 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:24:17.535623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:24:17.535625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:24:17.535842 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:24:17.535951 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 06:24:17.536346 (+ 395us) server_negotiation.cc:299] Negotiation successful
0807 06:24:17.536524 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 06:24:17.541105 32409 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:24:17.546079 32409 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:24:17.581014 623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:24:17.562223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:38203 (local address 127.0.0.1:41264)
0807 06:24:17.567012 (+ 4789us) negotiation.cc:107] Waiting for socket to connect
0807 06:24:17.567074 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0807 06:24:17.567217 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:24:17.569224 (+ 2007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:24:17.569230 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:24:17.569253 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:24:17.569426 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:24:17.569433 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.575160 (+ 5727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.575164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:24:17.576625 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:24:17.576636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.577932 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.577936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:24:17.578055 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:24:17.578093 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 06:24:17.579038 (+ 945us) client_negotiation.cc:240] Negotiation successful
0807 06:24:17.579175 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4594,"mutex_wait_us":1,"thread_start_us":150,"threads_started":1}
I20260807 06:24:17.583309 629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:24:17.565572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41854 (local address 127.15.254.2:36381)
0807 06:24:17.573007 (+ 7435us) server_negotiation.cc:206] Beginning negotiation
0807 06:24:17.573015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:24:17.573042 (+ 27us) server_negotiation.cc:407] Connection header received
0807 06:24:17.573121 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:24:17.573126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:24:17.573319 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:24:17.573474 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:24:17.573918 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.575235 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.581422 (+ 6187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.582362 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.582476 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:24:17.582622 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0807 06:24:17.583009 (+ 387us) server_negotiation.cc:299] Negotiation successful
0807 06:24:17.583120 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7305,"thread_start_us":87,"threads_started":1}
I20260807 06:24:17.583913 628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:24:17.565191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:36381 (local address 127.0.0.1:41854)
0807 06:24:17.572910 (+ 7719us) negotiation.cc:107] Waiting for socket to connect
0807 06:24:17.572923 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:24:17.572994 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:24:17.573640 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:24:17.573643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:24:17.573661 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:24:17.573762 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:24:17.573769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.575386 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.575390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:24:17.581254 (+ 5864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:24:17.581267 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.582691 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.582695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:24:17.582784 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:24:17.582808 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 06:24:17.582890 (+ 82us) client_negotiation.cc:240] Negotiation successful
0807 06:24:17.583687 (+ 797us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7657,"mutex_wait_us":5307,"thread_start_us":73,"threads_started":1}
I20260807 06:24:17.584432 624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:24:17.563210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41264 (local address 127.15.254.1:38203)
0807 06:24:17.566754 (+ 3544us) server_negotiation.cc:206] Beginning negotiation
0807 06:24:17.566762 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:24:17.567635 (+ 873us) server_negotiation.cc:407] Connection header received
0807 06:24:17.567722 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:24:17.567728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:24:17.568880 (+ 1152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:24:17.569040 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:24:17.569566 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.570737 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.576767 (+ 6030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.577600 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.577707 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:24:17.577867 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 06:24:17.584121 (+ 6254us) server_negotiation.cc:299] Negotiation successful
0807 06:24:17.584241 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3425,"thread_start_us":93,"threads_started":1}
I20260807 06:24:17.583725 626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:24:17.564033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:43685 (local address 127.0.0.1:55582)
0807 06:24:17.567451 (+ 3418us) negotiation.cc:107] Waiting for socket to connect
0807 06:24:17.567463 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 06:24:17.567537 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:24:17.572549 (+ 5012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:24:17.572552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:24:17.572606 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:24:17.572704 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:24:17.572710 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.579633 (+ 6923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.579637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:24:17.580791 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:24:17.580803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.582566 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.582569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:24:17.582656 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:24:17.582679 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 06:24:17.582738 (+ 59us) client_negotiation.cc:240] Negotiation successful
0807 06:24:17.583595 (+ 857us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3357,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
I20260807 06:24:17.587567 32602 tablet_service.cc:1467] Processing CreateTablet for tablet 630391c0f57e433ab3c824aa9bb8ce73 (DEFAULT_TABLE table=test-table [id=641b3d24336942efaeefa33c47b56b31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:24:17.587566 325 tablet_service.cc:1467] Processing CreateTablet for tablet 630391c0f57e433ab3c824aa9bb8ce73 (DEFAULT_TABLE table=test-table [id=641b3d24336942efaeefa33c47b56b31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:24:17.588867 325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 630391c0f57e433ab3c824aa9bb8ce73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:24:17.589764 32602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 630391c0f57e433ab3c824aa9bb8ce73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:24:17.599933 627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:24:17.564450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55582 (local address 127.15.254.3:43685)
0807 06:24:17.571963 (+ 7513us) server_negotiation.cc:206] Beginning negotiation
0807 06:24:17.571976 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 06:24:17.572003 (+ 27us) server_negotiation.cc:407] Connection header received
0807 06:24:17.572083 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:24:17.572089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:24:17.572320 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:24:17.573102 (+ 782us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:24:17.573249 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.579499 (+ 6250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.581425 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.582238 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.582350 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:24:17.582502 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 06:24:17.599618 (+ 17116us) server_negotiation.cc:299] Negotiation successful
0807 06:24:17.599764 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7384,"thread_start_us":107,"threads_started":1}
I20260807 06:24:17.603529 510 tablet_service.cc:1467] Processing CreateTablet for tablet 630391c0f57e433ab3c824aa9bb8ce73 (DEFAULT_TABLE table=test-table [id=641b3d24336942efaeefa33c47b56b31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:24:17.605952 510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 630391c0f57e433ab3c824aa9bb8ce73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:24:17.615684 630 tablet_bootstrap.cc:492] T 630391c0f57e433ab3c824aa9bb8ce73 P 1f4baaef44154f3a8728c36555a75a0d: Bootstrap starting.
I20260807 06:24:17.620311 630 tablet_bootstrap.cc:654] T 630391c0f57e433ab3c824aa9bb8ce73 P 1f4baaef44154f3a8728c36555a75a0d: Neither blocks nor log segments found. Creating new log.
I20260807 06:24:17.620904 632 tablet_bootstrap.cc:492] T 630391c0f57e433ab3c824aa9bb8ce73 P 833a4325a467456e92ee4de0bf47a196: Bootstrap starting.
I20260807 06:24:17.622474 633 tablet_bootstrap.cc:492] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998: Bootstrap starting.
I20260807 06:24:17.626098 633 tablet_bootstrap.cc:654] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998: Neither blocks nor log segments found. Creating new log.
I20260807 06:24:17.626905 630 log.cc:824] T 630391c0f57e433ab3c824aa9bb8ce73 P 1f4baaef44154f3a8728c36555a75a0d: Log is configured to *not* fsync() on all Append() calls
I20260807 06:24:17.628376 632 tablet_bootstrap.cc:654] T 630391c0f57e433ab3c824aa9bb8ce73 P 833a4325a467456e92ee4de0bf47a196: Neither blocks nor log segments found. Creating new log.
I20260807 06:24:17.630961 632 log.cc:824] T 630391c0f57e433ab3c824aa9bb8ce73 P 833a4325a467456e92ee4de0bf47a196: Log is configured to *not* fsync() on all Append() calls
I20260807 06:24:17.632310 633 log.cc:824] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998: Log is configured to *not* fsync() on all Append() calls
I20260807 06:24:17.634754 630 tablet_bootstrap.cc:492] T 630391c0f57e433ab3c824aa9bb8ce73 P 1f4baaef44154f3a8728c36555a75a0d: No bootstrap required, opened a new log
I20260807 06:24:17.635030 630 ts_tablet_manager.cc:1397] T 630391c0f57e433ab3c824aa9bb8ce73 P 1f4baaef44154f3a8728c36555a75a0d: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.000s
I20260807 06:24:17.642133 632 tablet_bootstrap.cc:492] T 630391c0f57e433ab3c824aa9bb8ce73 P 833a4325a467456e92ee4de0bf47a196: No bootstrap required, opened a new log
I20260807 06:24:17.642511 632 ts_tablet_manager.cc:1397] T 630391c0f57e433ab3c824aa9bb8ce73 P 833a4325a467456e92ee4de0bf47a196: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.002s
I20260807 06:24:17.639681 630 raft_consensus.cc:348] T 630391c0f57e433ab3c824aa9bb8ce73 P 1f4baaef44154f3a8728c36555a75a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4baaef44154f3a8728c36555a75a0d" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 38203 } } peers { permanent_uuid: "3cd0216079174a878b8d73097e522998" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43685 } } peers { permanent_uuid: "833a4325a467456e92ee4de0bf47a196" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 36381 } }
I20260807 06:24:17.647187 630 raft_consensus.cc:374] T 630391c0f57e433ab3c824aa9bb8ce73 P 1f4baaef44154f3a8728c36555a75a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:24:17.647313 630 raft_consensus.cc:729] T 630391c0f57e433ab3c824aa9bb8ce73 P 1f4baaef44154f3a8728c36555a75a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f4baaef44154f3a8728c36555a75a0d, State: Initialized, Role: FOLLOWER
I20260807 06:24:17.647953 630 consensus_queue.cc:260] T 630391c0f57e433ab3c824aa9bb8ce73 P 1f4baaef44154f3a8728c36555a75a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4baaef44154f3a8728c36555a75a0d" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 38203 } } peers { permanent_uuid: "3cd0216079174a878b8d73097e522998" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43685 } } peers { permanent_uuid: "833a4325a467456e92ee4de0bf47a196" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 36381 } }
I20260807 06:24:17.649015 632 raft_consensus.cc:348] T 630391c0f57e433ab3c824aa9bb8ce73 P 833a4325a467456e92ee4de0bf47a196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4baaef44154f3a8728c36555a75a0d" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 38203 } } peers { permanent_uuid: "3cd0216079174a878b8d73097e522998" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43685 } } peers { permanent_uuid: "833a4325a467456e92ee4de0bf47a196" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 36381 } }
I20260807 06:24:17.649369 632 raft_consensus.cc:374] T 630391c0f57e433ab3c824aa9bb8ce73 P 833a4325a467456e92ee4de0bf47a196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:24:17.649473 632 raft_consensus.cc:729] T 630391c0f57e433ab3c824aa9bb8ce73 P 833a4325a467456e92ee4de0bf47a196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833a4325a467456e92ee4de0bf47a196, State: Initialized, Role: FOLLOWER
I20260807 06:24:17.650025 632 consensus_queue.cc:260] T 630391c0f57e433ab3c824aa9bb8ce73 P 833a4325a467456e92ee4de0bf47a196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4baaef44154f3a8728c36555a75a0d" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 38203 } } peers { permanent_uuid: "3cd0216079174a878b8d73097e522998" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43685 } } peers { permanent_uuid: "833a4325a467456e92ee4de0bf47a196" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 36381 } }
I20260807 06:24:17.651069 632 ts_tablet_manager.cc:1428] T 630391c0f57e433ab3c824aa9bb8ce73 P 833a4325a467456e92ee4de0bf47a196: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260807 06:24:17.651437 633 tablet_bootstrap.cc:492] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998: No bootstrap required, opened a new log
I20260807 06:24:17.651497 412 heartbeater.cc:503] Master 127.15.254.62:36211 was elected leader, sending a full tablet report...
I20260807 06:24:17.651710 633 ts_tablet_manager.cc:1397] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.003s
I20260807 06:24:17.655915 633 raft_consensus.cc:348] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4baaef44154f3a8728c36555a75a0d" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 38203 } } peers { permanent_uuid: "3cd0216079174a878b8d73097e522998" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43685 } } peers { permanent_uuid: "833a4325a467456e92ee4de0bf47a196" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 36381 } }
I20260807 06:24:17.656292 633 raft_consensus.cc:374] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:24:17.656944 32696 heartbeater.cc:503] Master 127.15.254.62:36211 was elected leader, sending a full tablet report...
I20260807 06:24:17.667353 630 ts_tablet_manager.cc:1428] T 630391c0f57e433ab3c824aa9bb8ce73 P 1f4baaef44154f3a8728c36555a75a0d: Time spent starting tablet: real 0.032s user 0.007s sys 0.000s
I20260807 06:24:17.667766 633 raft_consensus.cc:729] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cd0216079174a878b8d73097e522998, State: Initialized, Role: FOLLOWER
I20260807 06:24:17.670255 633 consensus_queue.cc:260] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4baaef44154f3a8728c36555a75a0d" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 38203 } } peers { permanent_uuid: "3cd0216079174a878b8d73097e522998" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43685 } } peers { permanent_uuid: "833a4325a467456e92ee4de0bf47a196" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 36381 } }
I20260807 06:24:17.671947 600 heartbeater.cc:503] Master 127.15.254.62:36211 was elected leader, sending a full tablet report...
I20260807 06:24:17.671747 633 ts_tablet_manager.cc:1428] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998: Time spent starting tablet: real 0.020s user 0.005s sys 0.001s
W20260807 06:24:17.738749 601 tablet.cc:2369] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 07 06:24:17 dist-test-slave-44d2 krb5kdc[31774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083856, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for HTTP/127.15.254.20@KRBTEST.COM
W20260807 06:24:17.762331 413 tablet.cc:2369] T 630391c0f57e433ab3c824aa9bb8ce73 P 833a4325a467456e92ee4de0bf47a196: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:24:17.810935 640 raft_consensus.cc:482] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:24:17.811204 640 raft_consensus.cc:504] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4baaef44154f3a8728c36555a75a0d" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 38203 } } peers { permanent_uuid: "3cd0216079174a878b8d73097e522998" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43685 } } peers { permanent_uuid: "833a4325a467456e92ee4de0bf47a196" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 36381 } }
I20260807 06:24:17.812721 640 leader_election.cc:288] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f4baaef44154f3a8728c36555a75a0d (127.15.254.1:38203), 833a4325a467456e92ee4de0bf47a196 (127.15.254.2:36381)
I20260807 06:24:17.818004 603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:24:17.813115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:38203 (local address 127.15.254.3:49085)
0807 06:24:17.813263 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0807 06:24:17.813281 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:24:17.813389 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:24:17.813780 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:24:17.813786 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:24:17.813808 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:24:17.813957 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:24:17.813965 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.815107 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.815110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:24:17.816356 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:24:17.816382 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.817498 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.817502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:24:17.817619 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:24:17.817644 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 06:24:17.817722 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 06:24:17.817829 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260807 06:24:17.819128 624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:24:17.813225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:49085 (local address 127.15.254.1:38203)
0807 06:24:17.813355 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0807 06:24:17.813359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:24:17.813373 (+ 14us) server_negotiation.cc:407] Connection header received
0807 06:24:17.813541 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:24:17.813545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:24:17.813597 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:24:17.813665 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:24:17.814109 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.814918 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.816561 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.817215 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.817337 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:24:17.817415 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 06:24:17.818891 (+ 1476us) server_negotiation.cc:299] Negotiation successful
0807 06:24:17.818992 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 06:24:17.819840 32633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "630391c0f57e433ab3c824aa9bb8ce73" candidate_uuid: "3cd0216079174a878b8d73097e522998" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f4baaef44154f3a8728c36555a75a0d" is_pre_election: true
I20260807 06:24:17.820232 32633 raft_consensus.cc:2454] T 630391c0f57e433ab3c824aa9bb8ce73 P 1f4baaef44154f3a8728c36555a75a0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cd0216079174a878b8d73097e522998 in term 0.
I20260807 06:24:17.821039 440 leader_election.cc:302] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f4baaef44154f3a8728c36555a75a0d, 3cd0216079174a878b8d73097e522998; no voters:
I20260807 06:24:17.821417 640 raft_consensus.cc:2781] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:24:17.821527 640 raft_consensus.cc:482] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:24:17.821585 640 raft_consensus.cc:3037] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:24:17.823112 640 raft_consensus.cc:504] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4baaef44154f3a8728c36555a75a0d" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 38203 } } peers { permanent_uuid: "3cd0216079174a878b8d73097e522998" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43685 } } peers { permanent_uuid: "833a4325a467456e92ee4de0bf47a196" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 36381 } }
I20260807 06:24:17.823905 32633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "630391c0f57e433ab3c824aa9bb8ce73" candidate_uuid: "3cd0216079174a878b8d73097e522998" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f4baaef44154f3a8728c36555a75a0d"
I20260807 06:24:17.824048 32633 raft_consensus.cc:3037] T 630391c0f57e433ab3c824aa9bb8ce73 P 1f4baaef44154f3a8728c36555a75a0d [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:24:17.825377 32633 raft_consensus.cc:2454] T 630391c0f57e433ab3c824aa9bb8ce73 P 1f4baaef44154f3a8728c36555a75a0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd0216079174a878b8d73097e522998 in term 1.
I20260807 06:24:17.826059 440 leader_election.cc:302] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f4baaef44154f3a8728c36555a75a0d, 3cd0216079174a878b8d73097e522998; no voters:
I20260807 06:24:17.826299 640 leader_election.cc:288] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998 [CANDIDATE]: Term 1 election: Requested vote from peers 1f4baaef44154f3a8728c36555a75a0d (127.15.254.1:38203), 833a4325a467456e92ee4de0bf47a196 (127.15.254.2:36381)
I20260807 06:24:17.826413 640 raft_consensus.cc:2781] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:24:17.826515 640 raft_consensus.cc:686] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998 [term 1 LEADER]: Becoming Leader. State: Replica: 3cd0216079174a878b8d73097e522998, State: Running, Role: LEADER
I20260807 06:24:17.826910 640 consensus_queue.cc:237] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4baaef44154f3a8728c36555a75a0d" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 38203 } } peers { permanent_uuid: "3cd0216079174a878b8d73097e522998" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43685 } } peers { permanent_uuid: "833a4325a467456e92ee4de0bf47a196" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 36381 } }
I20260807 06:24:17.827068 603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:24:17.813431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:36381 (local address 127.15.254.3:37595)
0807 06:24:17.818196 (+ 4765us) negotiation.cc:107] Waiting for socket to connect
0807 06:24:17.818212 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:24:17.818318 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:24:17.819569 (+ 1251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:24:17.819572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:24:17.819597 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:24:17.819679 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:24:17.819684 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.821507 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.821511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:24:17.822619 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:24:17.822630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.826631 (+ 4001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.826635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:24:17.826743 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:24:17.826765 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 06:24:17.826846 (+ 81us) client_negotiation.cc:240] Negotiation successful
0807 06:24:17.826926 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4704,"mutex_wait_us":1,"thread_start_us":128,"threads_started":1}
I20260807 06:24:17.827157 629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:24:17.813863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:37595 (local address 127.15.254.2:36381)
0807 06:24:17.814070 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 06:24:17.814076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:24:17.818423 (+ 4347us) server_negotiation.cc:407] Connection header received
0807 06:24:17.818491 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:24:17.818494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:24:17.818546 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:24:17.818612 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:24:17.819832 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.820577 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.825726 (+ 5149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.826341 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.826493 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:24:17.826564 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 06:24:17.826914 (+ 350us) server_negotiation.cc:299] Negotiation successful
0807 06:24:17.827016 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 06:24:17.828032 356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "630391c0f57e433ab3c824aa9bb8ce73" candidate_uuid: "3cd0216079174a878b8d73097e522998" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833a4325a467456e92ee4de0bf47a196" is_pre_election: true
I20260807 06:24:17.828361 356 raft_consensus.cc:2454] T 630391c0f57e433ab3c824aa9bb8ce73 P 833a4325a467456e92ee4de0bf47a196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cd0216079174a878b8d73097e522998 in term 0.
I20260807 06:24:17.828049 355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "630391c0f57e433ab3c824aa9bb8ce73" candidate_uuid: "3cd0216079174a878b8d73097e522998" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833a4325a467456e92ee4de0bf47a196"
I20260807 06:24:17.828824 355 raft_consensus.cc:3037] T 630391c0f57e433ab3c824aa9bb8ce73 P 833a4325a467456e92ee4de0bf47a196 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:24:17.830309 355 raft_consensus.cc:2454] T 630391c0f57e433ab3c824aa9bb8ce73 P 833a4325a467456e92ee4de0bf47a196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd0216079174a878b8d73097e522998 in term 1.
I20260807 06:24:17.831624 32408 catalog_manager.cc:5495] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cd0216079174a878b8d73097e522998 (127.15.254.3). New cstate: current_term: 1 leader_uuid: "3cd0216079174a878b8d73097e522998" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4baaef44154f3a8728c36555a75a0d" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cd0216079174a878b8d73097e522998" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 43685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "833a4325a467456e92ee4de0bf47a196" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 36381 } health_report { overall_health: UNKNOWN } } }
I20260807 06:24:17.877665 627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:24:17.872166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55596 (local address 127.15.254.3:43685)
0807 06:24:17.872322 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0807 06:24:17.872328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:24:17.872347 (+ 19us) server_negotiation.cc:407] Connection header received
0807 06:24:17.872458 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:24:17.872462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:24:17.872561 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:24:17.872730 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:24:17.872983 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.873895 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.875173 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:24:17.875535 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:24:17.875633 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:24:17.876066 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:24:17.876710 (+ 644us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:24:17.876827 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 06:24:17.877270 (+ 443us) server_negotiation.cc:299] Negotiation successful
0807 06:24:17.877335 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 06:24:17.902513 355 raft_consensus.cc:1261] T 630391c0f57e433ab3c824aa9bb8ce73 P 833a4325a467456e92ee4de0bf47a196 [term 1 FOLLOWER]: Refusing update from remote peer 3cd0216079174a878b8d73097e522998: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:24:17.902643 32633 raft_consensus.cc:1261] T 630391c0f57e433ab3c824aa9bb8ce73 P 1f4baaef44154f3a8728c36555a75a0d [term 1 FOLLOWER]: Refusing update from remote peer 3cd0216079174a878b8d73097e522998: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:24:17.903533 646 consensus_queue.cc:1035] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833a4325a467456e92ee4de0bf47a196" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 36381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:24:17.904111 640 consensus_queue.cc:1035] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f4baaef44154f3a8728c36555a75a0d" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 06:24:17.907478 32697 tablet.cc:2369] T 630391c0f57e433ab3c824aa9bb8ce73 P 1f4baaef44154f3a8728c36555a75a0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:24:17.922263 649 mvcc.cc:205] Tried to move back new op lower bound from 7315799481908568064 to 7315799481657528320. Current Snapshot: MvccSnapshot[applied={T|T < 7315799481908568064}]
I20260807 06:24:17.933966 652 mvcc.cc:205] Tried to move back new op lower bound from 7315799481908568064 to 7315799481657528320. Current Snapshot: MvccSnapshot[applied={T|T < 7315799481908568064}]
I20260807 06:24:17.948006 650 mvcc.cc:205] Tried to move back new op lower bound from 7315799481908568064 to 7315799481657528320. Current Snapshot: MvccSnapshot[applied={T|T < 7315799481908568064}]
I20260807 06:24:17.981627 32408 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:24:17.981850 32408 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:24:17.984812 32408 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80009b60900c4391a142bb30893f9a46: Sending DeleteTablet for 3 replicas of tablet 630391c0f57e433ab3c824aa9bb8ce73
I20260807 06:24:17.986006 32602 tablet_service.cc:1514] Processing DeleteTablet for tablet 630391c0f57e433ab3c824aa9bb8ce73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:24:17 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:41264
I20260807 06:24:17.986270 510 tablet_service.cc:1514] Processing DeleteTablet for tablet 630391c0f57e433ab3c824aa9bb8ce73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:24:17 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:55582
I20260807 06:24:17.986797 666 tablet_replica.cc:330] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998: stopping tablet replica
I20260807 06:24:17.986842 325 tablet_service.cc:1514] Processing DeleteTablet for tablet 630391c0f57e433ab3c824aa9bb8ce73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:24:17 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:41854
I20260807 06:24:17.987192 666 raft_consensus.cc:2229] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:24:17.987535 669 tablet_replica.cc:330] T 630391c0f57e433ab3c824aa9bb8ce73 P 833a4325a467456e92ee4de0bf47a196: stopping tablet replica
I20260807 06:24:17.987649 666 raft_consensus.cc:2258] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:24:17.987895 669 raft_consensus.cc:2229] T 630391c0f57e433ab3c824aa9bb8ce73 P 833a4325a467456e92ee4de0bf47a196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:24:17.988270 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 32513
I20260807 06:24:17.988363 669 raft_consensus.cc:2258] T 630391c0f57e433ab3c824aa9bb8ce73 P 833a4325a467456e92ee4de0bf47a196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:24:17.996589 666 ts_tablet_manager.cc:1905] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:24:17.997903 669 ts_tablet_manager.cc:1905] T 630391c0f57e433ab3c824aa9bb8ce73 P 833a4325a467456e92ee4de0bf47a196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:24:17.999729 666 ts_tablet_manager.cc:1918] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:24:17.999943 666 log.cc:1196] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/630391c0f57e433ab3c824aa9bb8ce73
I20260807 06:24:18.000393 666 ts_tablet_manager.cc:1939] T 630391c0f57e433ab3c824aa9bb8ce73 P 3cd0216079174a878b8d73097e522998: Deleting consensus metadata
I20260807 06:24:18.000938 669 ts_tablet_manager.cc:1918] T 630391c0f57e433ab3c824aa9bb8ce73 P 833a4325a467456e92ee4de0bf47a196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:24:18.001060 669 log.cc:1196] T 630391c0f57e433ab3c824aa9bb8ce73 P 833a4325a467456e92ee4de0bf47a196: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/630391c0f57e433ab3c824aa9bb8ce73
I20260807 06:24:18.001430 669 ts_tablet_manager.cc:1939] T 630391c0f57e433ab3c824aa9bb8ce73 P 833a4325a467456e92ee4de0bf47a196: Deleting consensus metadata
I20260807 06:24:18.002213 32392 catalog_manager.cc:4841] TS 3cd0216079174a878b8d73097e522998 (127.15.254.3:43685): tablet 630391c0f57e433ab3c824aa9bb8ce73 (table test-table [id=641b3d24336942efaeefa33c47b56b31]) successfully deleted
I20260807 06:24:18.002743 32391 catalog_manager.cc:4841] TS 833a4325a467456e92ee4de0bf47a196 (127.15.254.2:36381): tablet 630391c0f57e433ab3c824aa9bb8ce73 (table test-table [id=641b3d24336942efaeefa33c47b56b31]) successfully deleted
W20260807 06:24:18.028301 32390 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.254.1:38203 (error 108)
I20260807 06:24:18.031901 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 32704
I20260807 06:24:18.065755 626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:24:18.065244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:38203 (local address 127.0.0.1:41272)
0807 06:24:18.065508 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0807 06:24:18.065617 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.1:38203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
W20260807 06:24:18.066044 32390 catalog_manager.cc:4584] TS 1f4baaef44154f3a8728c36555a75a0d (127.15.254.1:38203): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 630391c0f57e433ab3c824aa9bb8ce73: Network error: Client connection negotiation failed: client connection to 127.15.254.1:38203: connect: Connection refused (error 111)
I20260807 06:24:18.067056 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 421
I20260807 06:24:18.111251 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 32366
I20260807 06:24:18.125623 16376 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260807 06:24:18.480854 16376 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260807 06:24:18.481000 16376 mini_ranger.cc:66] Stopping Ranger...
I20260807 06:24:18.837051 16376 mini_ranger.cc:68] Stopped Ranger
2026-08-07 06:24:18.837 UTC [31816] LOG: received smart shutdown request
2026-08-07 06:24:18.841 UTC [31816] LOG: background worker "logical replication launcher" (PID 31828) exited with exit code 1
2026-08-07 06:24:18.841 UTC [31822] LOG: shutting down
2026-08-07 06:24:19.057 UTC [31816] LOG: database system is shut down
2026-08-07T06:24:19Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51468 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:24:19 dist-test-slave-44d2 krb5kdc[685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:24:19 dist-test-slave-44d2 krb5kdc[685](info): set up 2 sockets
Aug 07 06:24:19 dist-test-slave-44d2 krb5kdc[685](info): commencing operation
krb5kdc: starting...
W20260807 06:24:20.443388 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.223s user 0.005s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:24:20 dist-test-slave-44d2 krb5kdc[685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:24:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:24:20Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.254.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.254.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.254.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/rangeradmin_127.15.254.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.254.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/rangeradmin_127.15.254.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.254.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.254.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.254.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/rangerlookup_127.15.254.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.254.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/rangerlookup_127.15.254.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.254.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.254.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.254.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/HTTP_127.15.254.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.254.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/HTTP_127.15.254.20@KRBTEST.COM.keytab.
I20260807 06:24:20.660049 16376 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskZSVWny/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/postgres -l logfile start
2026-08-07 06:24:23.614 UTC [721] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-07 06:24:23.614 UTC [721] LOG: listening on IPv4 address "127.15.254.20", port 53089
2026-08-07 06:24:23.618 UTC [721] LOG: listening on Unix socket "/tmp/.s.PGSQL.53089"
2026-08-07 06:24:23.641 UTC [726] LOG: database system was shut down at 2026-08-07 06:24:21 UTC
2026-08-07 06:24:23.646 UTC [721] LOG: database system is ready to accept connections
I20260807 06:24:24.785881 16376 mini_postgres.cc:96] Postgres bound to 53089
2026-08-07 06:24:24.793 UTC [737] FATAL: database "slave" does not exist
127.15.254.20:53089 - accepting connections
I20260807 06:24:24.794466 16376 mini_ranger.cc:161] Starting Ranger...
I20260807 06:24:24.811707 16376 mini_ranger.cc:84] Created miniranger Postgres user
I20260807 06:24:25.383009 16376 mini_ranger.cc:87] Created ranger Postgres database
I20260807 06:24:25.383113 16376 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ranger-admin
2026-08-07 06:24:25,683 [I] DB FLAVOR :POSTGRES
2026-08-07 06:24:25,684 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 06:24:25,684 [I] Checking connection..
2026-08-07 06:24:25,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:53089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-07 06:24:25,957 [I] Checking connection passed.
2026-08-07 06:24:25,957 [I] --------- Verifying version history table ---------
2026-08-07 06:24:25,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:53089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 06:24:26,222 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 06:24:26,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:53089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 06:24:26,500 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 06:24:26,500 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-07 06:24:26,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:53089/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-07 06:24:26.782 UTC [823] WARNING: there is no transaction in progress
2026-08-07 06:24:26,792 [I] create_dbversion_catalog.sql file imported successfully
2026-08-07 06:24:26,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:53089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 06:24:27,052 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-07 06:24:27,053 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 06:24:27,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:53089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-07 06:24:27,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:53089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-07 06:24:27,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:53089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-44d2.c.gcp-upstream.internal','N') ;"
2026-08-07 06:24:27,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:53089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 06:24:28,096 [I] Table x_portal_user does not exist in database ranger
2026-08-07 06:24:28,096 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:53089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 06:24:28,378 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-07 06:24:28,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:53089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 06:24:28,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:53089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 06:24:28,917 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-07 06:24:28,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:53089/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-07 06:24:30.013 UTC [968] WARNING: there is no transaction in progress
2026-08-07 06:24:30.032 UTC [968] WARNING: there is no transaction in progress
2026-08-07 06:24:30.050 UTC [968] WARNING: there is no transaction in progress
2026-08-07 06:24:30.068 UTC [968] WARNING: there is no transaction in progress
2026-08-07 06:24:30.085 UTC [968] WARNING: there is no transaction in progress
2026-08-07 06:24:30.104 UTC [968] WARNING: there is no transaction in progress
2026-08-07 06:24:30.160 UTC [968] WARNING: there is no transaction in progress
2026-08-07 06:24:30.168 UTC [968] WARNING: there is no transaction in progress
2026-08-07 06:24:30.183 UTC [968] WARNING: there is no transaction in progress
2026-08-07 06:24:30.195 UTC [968] WARNING: there is no transaction in progress
2026-08-07 06:24:30.205 UTC [968] WARNING: there is no transaction in progress
2026-08-07 06:24:30.219 UTC [968] WARNING: there is no transaction in progress
2026-08-07 06:24:30.230 UTC [968] WARNING: there is no transaction in progress
2026-08-07 06:24:30.237 UTC [968] WARNING: there is no transaction in progress
2026-08-07 06:24:30.245 UTC [968] WARNING: there is no transaction in progress
2026-08-07 06:24:30.253 UTC [968] WARNING: there is no transaction in progress
2026-08-07 06:24:31,003 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-07 06:24:31,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:53089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-07 06:24:31,270 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-07 06:24:31,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:53089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 06:24:31,564 [I] Table x_portal_user already exists in database 'ranger'
2026-08-07 06:24:31,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:53089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 06:24:31,855 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-07 06:24:31,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:53089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 06:24:32,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:53089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 06:24:32,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:53089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-44d2.c.gcp-upstream.internal';"
2026-08-07 06:24:32,625 [I] CORE_DB_SCHEMA import status has been updated
2026-08-07 06:24:32,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:53089/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-07 06:24:32,921 [I] DB_PATCHES have already been applied
I20260807 06:24:32.927436 16376 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ranger-admin:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskZSVWny/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260807 06:24:32.927516 16376 mini_ranger.cc:193] Using host: 127.15.254.20
I20260807 06:24:32.932744 16376 mini_ranger.cc:227] Ranger admin URL: http://127.15.254.20:37945
Aug 07, 2026 6:24:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 6:24:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 6:24:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ranger-admin/ews/webapp
Aug 07, 2026 6:24:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 07, 2026 6:24:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 07, 2026 6:24:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ranger-admin/ews/webapp].
Aug 07, 2026 6:24:33 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260807 06:24:34.242969 16376 mini_ranger.cc:160] Time spent starting Ranger: real 9.449s user 0.005s sys 0.009s
Aug 07, 2026 6:24:56 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 07, 2026 6:24:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 07, 2026 6:24:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 07, 2026 6:24:56 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 07, 2026 6:24:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 07, 2026 6:24:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 07, 2026 6:24:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 07, 2026 6:24:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 07, 2026 6:24:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 07, 2026 6:24:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 07, 2026 6:24:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 07, 2026 6:24:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 07, 2026 6:24:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 07, 2026 6:24:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 07, 2026 6:24:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 07, 2026 6:24:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 07, 2026 6:24:56 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 07, 2026 6:24:57 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260807 06:24:58.131861 16376 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.254.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.254.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.254.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/rangerkms_127.15.254.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.254.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/rangerkms_127.15.254.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.254.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.254.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.254.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/HTTP_127.15.254.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.254.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/HTTP_127.15.254.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 07 06:24:58 dist-test-slave-44d2 krb5kdc[685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083898, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260807 06:24:58.241123 16376 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260807 06:24:58.252445 16376 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260807 06:24:59.294525 16376 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260807 06:24:59.294627 16376 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ranger-kms
I20260807 06:24:59.294651 16376 mini_ranger_kms.cc:227] Using postgres at 127.15.254.20:53089
2026-08-07 06:24:59,495 [I] DB FLAVOR :POSTGRES
2026-08-07 06:24:59,495 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 06:24:59,495 [I] Checking connection
2026-08-07 06:24:59,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:53089/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-07 06:24:59,778 [I] connection success
2026-08-07 06:24:59,779 [I] --------- Verifying Ranger DB tables ---------
2026-08-07 06:24:59,779 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-07 06:24:59,779 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:53089/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-07 06:25:00,055 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-07 06:25:00,056 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 06:25:00,056 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-07 06:25:00,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.254.20:53089/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-07 06:25:00,374 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260807 06:25:00.905916 16376 mini_ranger_kms.cc:310] Created kms service
I20260807 06:25:01.125506 16376 mini_ranger_kms.cc:326] Created kudu user
I20260807 06:25:01.182278 16376 mini_ranger_kms.cc:343] Created rangerkms user
Aug 07 06:25:01 dist-test-slave-44d2 krb5kdc[685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083898, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.254.20@KRBTEST.COM
I20260807 06:25:01.635784 16376 mini_ranger_kms.cc:384] Added ranger policy
I20260807 06:25:01.635942 16376 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ranger-kms:/tmp/dist-test-taskZSVWny/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskZSVWny/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260807 06:25:01.635963 16376 mini_ranger_kms.cc:240] Using host: 127.15.254.20
I20260807 06:25:01.641938 16376 mini_ranger_kms.cc:276] Ranger KMS PID: 1232
I20260807 06:25:01.642062 16376 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.254.20:45981
Aug 07, 2026 6:25:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 6:25:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 6:25:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 07, 2026 6:25:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 07, 2026 6:25:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ranger-kms/ews/webapp].
Aug 07, 2026 6:25:03 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260807 06:25:04.642570 16376 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.401s user 0.003s sys 0.180s
I20260807 06:25:04.642710 16376 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260807 06:25:04.642781 16376 mini_ranger_kms.cc:401] 127.15.254.20:45981/kms/v1/keys
Aug 07 06:25:04 dist-test-slave-44d2 krb5kdc[685](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786083904, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.254.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 07 06:25:05 dist-test-slave-44d2 krb5kdc[685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083904, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.254.20@KRBTEST.COM for HTTP/127.15.254.20@KRBTEST.COM
Aug 07 06:25:05 dist-test-slave-44d2 krb5kdc[685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083904, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.254.20@KRBTEST.COM for HTTP/127.15.254.20@KRBTEST.COM
Aug 07 06:25:08 dist-test-slave-44d2 krb5kdc[685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:25:08.653017 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:35437
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:35203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:35437
--ranger_config_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.254.20:45981/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:25:08.792469 1298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:08.792966 1298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:08.793077 1298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:08.797333 1298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:25:08.797426 1298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:08.797468 1298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:08.797528 1298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:25:08.797565 1298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:25:08.803277 1298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.254.20:45981/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:35437
--ranger_config_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:35437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:08.804626 1298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:08.805763 1298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:08.816751 1308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:08.816866 1307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:08.816996 1298 server_base.cc:1034] running on GCE node
W20260807 06:25:08.816735 1311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:08.817797 1298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:08.819111 1298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:08.820317 1298 hybrid_clock.cc:648] HybridClock initialized: now 1786083908820303 us; error 33 us; skew 500 ppm
Aug 07 06:25:08 dist-test-slave-44d2 krb5kdc[685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083908, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:08.827905 1298 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:25:08.829178 1298 webserver.cc:460] Webserver started at http://127.15.254.62:33417/ using document root <none> and password file <none>
I20260807 06:25:08.829885 1298 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:08.829964 1298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:25:08.830195 1298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 06:25:08 dist-test-slave-44d2 krb5kdc[685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083908, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for HTTP/127.15.254.20@KRBTEST.COM
I20260807 06:25:08.880712 1298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "e54b0808633548808dd1fcf36d51c7d4"
format_stamp: "Formatted at 2026-08-07 06:25:08 on dist-test-slave-44d2"
server_key: "c8d52bb5ef32e9974af925019e2ec73d"
server_key_iv: "64ae2f78ab316c88f4d9ccdab331d2c8"
server_key_version: "kuduclusterkey@0"
I20260807 06:25:08.881374 1298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "e54b0808633548808dd1fcf36d51c7d4"
format_stamp: "Formatted at 2026-08-07 06:25:08 on dist-test-slave-44d2"
server_key: "c8d52bb5ef32e9974af925019e2ec73d"
server_key_iv: "64ae2f78ab316c88f4d9ccdab331d2c8"
server_key_version: "kuduclusterkey@0"
I20260807 06:25:08.885588 1298 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 06:25:08.905038 1324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:08.906654 1298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:25:08.906811 1298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "e54b0808633548808dd1fcf36d51c7d4"
format_stamp: "Formatted at 2026-08-07 06:25:08 on dist-test-slave-44d2"
server_key: "c8d52bb5ef32e9974af925019e2ec73d"
server_key_iv: "64ae2f78ab316c88f4d9ccdab331d2c8"
server_key_version: "kuduclusterkey@0"
I20260807 06:25:08.906972 1298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:08.929219 1298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:08.932770 1298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:08.933012 1298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:08.942094 1298 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:35437
I20260807 06:25:08.942109 1402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:35437 every 8 connection(s)
I20260807 06:25:08.943195 1298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:25:08.944016 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 1298
I20260807 06:25:08.944185 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:25:08.944542 16376 external_mini_cluster.cc:1428] Setting key e2ff019fc518c3bd60d30f2bb404ed17
I20260807 06:25:08.946681 1403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:25:08.952399 1403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4: Bootstrap starting.
Aug 07 06:25:08 dist-test-slave-44d2 krb5kdc[685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:25:08.954907 1403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4: Neither blocks nor log segments found. Creating new log.
I20260807 06:25:08.955665 1403 log.cc:824] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4: Log is configured to *not* fsync() on all Append() calls
I20260807 06:25:08.957649 1403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4: No bootstrap required, opened a new log
I20260807 06:25:08.960326 1406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:08.946054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42628 (local address 127.15.254.62:35437)
0807 06:25:08.946533 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:08.946550 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:08.946588 (+ 38us) server_negotiation.cc:407] Connection header received
0807 06:25:08.947369 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:08.947389 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:08.947732 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:08.948088 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:08.949001 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:08.950026 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:08.950934 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:08.951410 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:08.951592 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:08.954561 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:08.954568 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:08.954582 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:08.954612 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:08.957207 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:08.957754 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:08.957762 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:08.957770 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:08.957925 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:08.958260 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:08.958265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:08.958267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:08.958718 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:08.958962 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:08.959300 (+ 338us) server_negotiation.cc:299] Negotiation successful
0807 06:25:08.959605 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":153,"threads_started":1}
I20260807 06:25:08.960906 1403 raft_consensus.cc:348] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54b0808633548808dd1fcf36d51c7d4" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 35437 } }
I20260807 06:25:08.961193 1403 raft_consensus.cc:374] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:25:08.961246 1403 raft_consensus.cc:729] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e54b0808633548808dd1fcf36d51c7d4, State: Initialized, Role: FOLLOWER
I20260807 06:25:08.961688 1403 consensus_queue.cc:260] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54b0808633548808dd1fcf36d51c7d4" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 35437 } }
I20260807 06:25:08.961814 1403 raft_consensus.cc:388] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:25:08.961897 1403 raft_consensus.cc:482] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:25:08.962006 1403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:25:08.963074 1403 raft_consensus.cc:504] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54b0808633548808dd1fcf36d51c7d4" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 35437 } }
I20260807 06:25:08.963414 1403 leader_election.cc:302] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e54b0808633548808dd1fcf36d51c7d4; no voters:
I20260807 06:25:08.963721 1403 leader_election.cc:288] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:25:08.963946 1408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:25:08.964212 1408 raft_consensus.cc:686] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [term 1 LEADER]: Becoming Leader. State: Replica: e54b0808633548808dd1fcf36d51c7d4, State: Running, Role: LEADER
I20260807 06:25:08.964596 1408 consensus_queue.cc:237] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54b0808633548808dd1fcf36d51c7d4" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 35437 } }
I20260807 06:25:08.965022 1403 sys_catalog.cc:564] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:25:08.966353 1409 sys_catalog.cc:455] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e54b0808633548808dd1fcf36d51c7d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54b0808633548808dd1fcf36d51c7d4" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 35437 } } }
I20260807 06:25:08.966840 1409 sys_catalog.cc:458] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [sys.catalog]: This master's current role is: LEADER
I20260807 06:25:08.966711 1411 sys_catalog.cc:455] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e54b0808633548808dd1fcf36d51c7d4. Latest consensus state: current_term: 1 leader_uuid: "e54b0808633548808dd1fcf36d51c7d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54b0808633548808dd1fcf36d51c7d4" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 35437 } } }
I20260807 06:25:08.967203 1411 sys_catalog.cc:458] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [sys.catalog]: This master's current role is: LEADER
I20260807 06:25:08.967528 1403 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 06:25:08.967762 1428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:25:08.975474 1428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:25:08.983680 1428 catalog_manager.cc:1347] Generated new cluster ID: 6059d08ee70c445980b7bad1dd9c5bc2
I20260807 06:25:08.983764 1428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:25:09.015324 1428 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:25:09.016821 1428 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:25:09.031780 1428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4: Generated new TSK 0
I20260807 06:25:09.032785 1428 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
I20260807 06:25:09.765713 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.254.20:45981/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:0
--local_ip_for_outbound_sockets=127.15.254.1
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:35437
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:35203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:25:09.929746 1461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:09.930071 1461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:09.930195 1461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:09.935427 1461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:09.935516 1461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:09.935618 1461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:25:09.944070 1461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.254.20:45981/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:35437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:09.945720 1461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:09.947139 1461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:09.959057 1476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:09.959486 1461 server_base.cc:1034] running on GCE node
W20260807 06:25:09.959079 1474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:09.959795 1473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:09.960356 1461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:09.961225 1461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:09.962433 1461 hybrid_clock.cc:648] HybridClock initialized: now 1786083909962396 us; error 59 us; skew 500 ppm
Aug 07 06:25:09 dist-test-slave-44d2 krb5kdc[685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083909, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:09.966526 1461 init.cc:377] Logged in from keytab as kudu/127.15.254.1@KRBTEST.COM (short username kudu)
I20260807 06:25:09.967856 1461 webserver.cc:460] Webserver started at http://127.15.254.1:45917/ using document root <none> and password file <none>
I20260807 06:25:09.968703 1461 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:09.968811 1461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:25:09.969156 1461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 06:25:09 dist-test-slave-44d2 krb5kdc[685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083909, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for HTTP/127.15.254.20@KRBTEST.COM
I20260807 06:25:10.000428 1461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/data/instance:
uuid: "e688f4fbe7244ee6a61c824db42dd62d"
format_stamp: "Formatted at 2026-08-07 06:25:09 on dist-test-slave-44d2"
server_key: "2158e5d383228c9e0a2e7e5e9cb9e252"
server_key_iv: "61b9360ecb491e8f8b5d35539c22a05e"
server_key_version: "kuduclusterkey@0"
I20260807 06:25:10.003348 1461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance:
uuid: "e688f4fbe7244ee6a61c824db42dd62d"
format_stamp: "Formatted at 2026-08-07 06:25:09 on dist-test-slave-44d2"
server_key: "2158e5d383228c9e0a2e7e5e9cb9e252"
server_key_iv: "61b9360ecb491e8f8b5d35539c22a05e"
server_key_version: "kuduclusterkey@0"
I20260807 06:25:10.015909 1461 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.000s
I20260807 06:25:10.039348 1488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:10.041213 1461 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260807 06:25:10.041368 1461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "e688f4fbe7244ee6a61c824db42dd62d"
format_stamp: "Formatted at 2026-08-07 06:25:09 on dist-test-slave-44d2"
server_key: "2158e5d383228c9e0a2e7e5e9cb9e252"
server_key_iv: "61b9360ecb491e8f8b5d35539c22a05e"
server_key_version: "kuduclusterkey@0"
I20260807 06:25:10.041512 1461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:10.059653 1461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:10.064108 1461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:10.064455 1461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:10.065361 1461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:25:10.066771 1461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:25:10.066898 1461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:10.067040 1461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:25:10.067135 1461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:10.091727 1461 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:37419
I20260807 06:25:10.092715 1641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:37419 every 8 connection(s)
I20260807 06:25:10.093158 1461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:25:10.102371 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 1461
I20260807 06:25:10.102538 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance
I20260807 06:25:10.102917 16376 external_mini_cluster.cc:1428] Setting key 0b72cff9a908a6b420045474b693c878
Aug 07 06:25:10 dist-test-slave-44d2 krb5kdc[685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083909, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:25:10.113159 1645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:10.095737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:36981 (local address 127.15.254.62:35437)
0807 06:25:10.096092 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:10.096099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:10.098059 (+ 1960us) server_negotiation.cc:407] Connection header received
0807 06:25:10.099431 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:10.099437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:10.099517 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:10.099636 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:10.101391 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:10.102290 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:10.104171 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:10.104563 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:10.104643 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:10.108216 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:10.108221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:10.108223 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:10.108252 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:10.110255 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:10.110938 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:10.110942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:10.110944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:10.111058 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:10.111511 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:10.111515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:10.111517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:10.111749 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:10.111824 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:10.112749 (+ 925us) server_negotiation.cc:299] Negotiation successful
0807 06:25:10.112902 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":108,"threads_started":1}
I20260807 06:25:10.115142 1646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:10.096246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:35437 (local address 127.15.254.1:36981)
0807 06:25:10.097773 (+ 1527us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:10.097842 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:10.099125 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:10.099866 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:10.099876 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:10.100416 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:10.101186 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:10.101222 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:10.102453 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:10.102456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:10.104011 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:10.104024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:10.104869 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:10.104874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:10.105072 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:10.105793 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:10.105815 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:10.107993 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:10.110411 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:10.110417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:10.110434 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:10.110794 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:10.111203 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:10.111206 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:10.111208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:10.111394 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:10.111932 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:10.111939 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:10.112386 (+ 447us) client_negotiation.cc:769] Sending connection context
0807 06:25:10.113625 (+ 1239us) client_negotiation.cc:240] Negotiation successful
0807 06:25:10.114008 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1271,"thread_start_us":312,"threads_started":1}
I20260807 06:25:10.117264 1642 heartbeater.cc:347] Connected to a master server at 127.15.254.62:35437
I20260807 06:25:10.117599 1642 heartbeater.cc:464] Registering TS with master...
Aug 07 06:25:10 dist-test-slave-44d2 krb5kdc[685](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786083910, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:10.118257 1642 heartbeater.cc:511] Master 127.15.254.62:35437 requested a full tablet report, sending...
I20260807 06:25:10.120571 1348 ts_manager.cc:194] Registered new tserver with Master: e688f4fbe7244ee6a61c824db42dd62d (127.15.254.1:37419)
I20260807 06:25:10.122232 1348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.1@KRBTEST.COM'} at 127.15.254.1:36981
WARNING: no policy specified for kudu/127.15.254.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
I20260807 06:25:10.154026 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.254.20:45981/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:0
--local_ip_for_outbound_sockets=127.15.254.2
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:35437
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:35203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:25:10.298936 1650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:10.299265 1650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:10.299394 1650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:10.304419 1650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:10.304531 1650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:10.304651 1650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:25:10.312613 1650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.254.20:45981/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:35437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:10.314388 1650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:10.315882 1650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:10.330629 1662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:10.330677 1660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:10.331148 1659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:10.332445 1650 server_base.cc:1034] running on GCE node
I20260807 06:25:10.332973 1650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:10.333761 1650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:10.334960 1650 hybrid_clock.cc:648] HybridClock initialized: now 1786083910334924 us; error 52 us; skew 500 ppm
Aug 07 06:25:10 dist-test-slave-44d2 krb5kdc[685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083910, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:10.339514 1650 init.cc:377] Logged in from keytab as kudu/127.15.254.2@KRBTEST.COM (short username kudu)
I20260807 06:25:10.340987 1650 webserver.cc:460] Webserver started at http://127.15.254.2:33421/ using document root <none> and password file <none>
I20260807 06:25:10.341954 1650 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:10.342079 1650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:25:10.342494 1650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 06:25:10 dist-test-slave-44d2 krb5kdc[685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083910, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for HTTP/127.15.254.20@KRBTEST.COM
I20260807 06:25:10.368016 1650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/data/instance:
uuid: "420fe814e5da47cda4cc863ba8fd3401"
format_stamp: "Formatted at 2026-08-07 06:25:10 on dist-test-slave-44d2"
server_key: "69e31988843f53c2f0beb1061bc8cf6a"
server_key_iv: "07eaf7d99946a3986cee1343ab16859a"
server_key_version: "kuduclusterkey@0"
I20260807 06:25:10.368784 1650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance:
uuid: "420fe814e5da47cda4cc863ba8fd3401"
format_stamp: "Formatted at 2026-08-07 06:25:10 on dist-test-slave-44d2"
server_key: "69e31988843f53c2f0beb1061bc8cf6a"
server_key_iv: "07eaf7d99946a3986cee1343ab16859a"
server_key_version: "kuduclusterkey@0"
I20260807 06:25:10.374037 1650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 06:25:10.391930 1670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:10.394099 1650 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260807 06:25:10.394337 1650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "420fe814e5da47cda4cc863ba8fd3401"
format_stamp: "Formatted at 2026-08-07 06:25:10 on dist-test-slave-44d2"
server_key: "69e31988843f53c2f0beb1061bc8cf6a"
server_key_iv: "07eaf7d99946a3986cee1343ab16859a"
server_key_version: "kuduclusterkey@0"
I20260807 06:25:10.394575 1650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:10.429636 1650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:10.433490 1650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:10.433722 1650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:10.434540 1650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:25:10.435730 1650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:25:10.435802 1650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:10.435858 1650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:25:10.435889 1650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:10.458379 1650 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:34607
I20260807 06:25:10.458468 1827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:34607 every 8 connection(s)
I20260807 06:25:10.459607 1650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:25:10.472070 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 1650
I20260807 06:25:10.472252 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance
I20260807 06:25:10.472646 16376 external_mini_cluster.cc:1428] Setting key 43c933a2ae1579e8da949b2c31e2e540
Aug 07 06:25:10 dist-test-slave-44d2 krb5kdc[685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083910, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:25:10.485075 1645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:10.462602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:50325 (local address 127.15.254.62:35437)
0807 06:25:10.462796 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:10.462801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:10.463930 (+ 1129us) server_negotiation.cc:407] Connection header received
0807 06:25:10.465657 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:10.465660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:10.465708 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:10.465775 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:10.467436 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:10.468290 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:10.470104 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:10.470484 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:10.470560 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:10.480117 (+ 9557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:10.480121 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:10.480124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:10.480155 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:10.482128 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:10.482912 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:10.482916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:10.482918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:10.483016 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:10.483479 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:10.483483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:10.483485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:10.483734 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:10.483825 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:10.484707 (+ 882us) server_negotiation.cc:299] Negotiation successful
0807 06:25:10.484879 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 06:25:10.487274 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:10.463009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:35437 (local address 127.15.254.2:50325)
0807 06:25:10.463684 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:10.463771 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:10.465383 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:10.466003 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:10.466014 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:10.466478 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:10.467234 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:10.467257 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:10.468521 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:10.468525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:10.469948 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:10.469960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:10.470798 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:10.470801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:10.471027 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:10.478080 (+ 7053us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:10.478101 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:10.479867 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:10.482295 (+ 2428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:10.482304 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:10.482326 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:10.482753 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:10.483138 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:10.483141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:10.483143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:10.483354 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:10.483913 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:10.483921 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:10.484315 (+ 394us) client_negotiation.cc:769] Sending connection context
0807 06:25:10.485602 (+ 1287us) client_negotiation.cc:240] Negotiation successful
0807 06:25:10.486044 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":178,"threads_started":1}
I20260807 06:25:10.489080 1828 heartbeater.cc:347] Connected to a master server at 127.15.254.62:35437
I20260807 06:25:10.489373 1828 heartbeater.cc:464] Registering TS with master...
I20260807 06:25:10.490022 1828 heartbeater.cc:511] Master 127.15.254.62:35437 requested a full tablet report, sending...
I20260807 06:25:10.491328 1348 ts_manager.cc:194] Registered new tserver with Master: 420fe814e5da47cda4cc863ba8fd3401 (127.15.254.2:34607)
I20260807 06:25:10.492128 1348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.2@KRBTEST.COM'} at 127.15.254.2:50325
WARNING: no policy specified for kudu/127.15.254.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
I20260807 06:25:10.522258 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.254.20:45981/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:0
--local_ip_for_outbound_sockets=127.15.254.3
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:35437
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:35203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 06:25:10 dist-test-slave-44d2 krb5kdc[685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083910, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for HTTP/127.15.254.20@KRBTEST.COM
W20260807 06:25:10.785455 1833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:10.785955 1833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:10.786063 1833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:10.791868 1833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:10.791963 1833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:10.792047 1833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:25:10.799683 1833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.254.20:45981/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:35437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:10.801246 1833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:10.802732 1833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:10.816646 1839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:10.816651 1838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:10.817148 1833 server_base.cc:1034] running on GCE node
W20260807 06:25:10.817449 1841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:10.818049 1833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:10.819031 1833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:10.820262 1833 hybrid_clock.cc:648] HybridClock initialized: now 1786083910820198 us; error 73 us; skew 500 ppm
Aug 07 06:25:10 dist-test-slave-44d2 krb5kdc[685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083910, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:10.824697 1833 init.cc:377] Logged in from keytab as kudu/127.15.254.3@KRBTEST.COM (short username kudu)
I20260807 06:25:10.825996 1833 webserver.cc:460] Webserver started at http://127.15.254.3:32801/ using document root <none> and password file <none>
I20260807 06:25:10.826808 1833 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:10.826913 1833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:25:10.827168 1833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 06:25:10 dist-test-slave-44d2 krb5kdc[685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083910, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for HTTP/127.15.254.20@KRBTEST.COM
I20260807 06:25:10.842676 1833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/data/instance:
uuid: "68be4dc9ed0b40b4bc2f84e9b35682d8"
format_stamp: "Formatted at 2026-08-07 06:25:10 on dist-test-slave-44d2"
server_key: "b51ab230a7aa16c19d229b33a1a83e68"
server_key_iv: "bd22178549a0590bb6fa5d7d42ee4042"
server_key_version: "kuduclusterkey@0"
I20260807 06:25:10.843466 1833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance:
uuid: "68be4dc9ed0b40b4bc2f84e9b35682d8"
format_stamp: "Formatted at 2026-08-07 06:25:10 on dist-test-slave-44d2"
server_key: "b51ab230a7aa16c19d229b33a1a83e68"
server_key_iv: "bd22178549a0590bb6fa5d7d42ee4042"
server_key_version: "kuduclusterkey@0"
I20260807 06:25:10.850045 1833 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260807 06:25:10.868045 1854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:10.869976 1833 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260807 06:25:10.870182 1833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "68be4dc9ed0b40b4bc2f84e9b35682d8"
format_stamp: "Formatted at 2026-08-07 06:25:10 on dist-test-slave-44d2"
server_key: "b51ab230a7aa16c19d229b33a1a83e68"
server_key_iv: "bd22178549a0590bb6fa5d7d42ee4042"
server_key_version: "kuduclusterkey@0"
I20260807 06:25:10.870396 1833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:10.900218 1833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:10.904024 1833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:10.904279 1833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:10.905100 1833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:25:10.906214 1833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:25:10.906316 1833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:10.906420 1833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:25:10.906489 1833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:10.925348 1833 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:37133
I20260807 06:25:10.926676 1833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:25:10.929131 2004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:37133 every 8 connection(s)
I20260807 06:25:10.934980 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 1833
I20260807 06:25:10.935130 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance
I20260807 06:25:10.935504 16376 external_mini_cluster.cc:1428] Setting key 9f30981a8d803cebb708b1198b821442
Aug 07 06:25:10 dist-test-slave-44d2 krb5kdc[685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083910, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:25:10.951903 1645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:10.931759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:43567 (local address 127.15.254.62:35437)
0807 06:25:10.931923 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:10.931929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:10.933792 (+ 1863us) server_negotiation.cc:407] Connection header received
0807 06:25:10.936602 (+ 2810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:10.936607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:10.936672 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:10.936735 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:10.938696 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:10.939503 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:10.941315 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:10.941589 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:10.941722 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:10.946828 (+ 5106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:10.946834 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:10.946837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:10.946872 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:10.949053 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:10.949790 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:10.949794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:10.949796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:10.949895 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:10.950382 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:10.950386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:10.950388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:10.950689 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:10.950787 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:10.951595 (+ 808us) server_negotiation.cc:299] Negotiation successful
0807 06:25:10.951736 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 06:25:10.953825 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:10.932267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:35437 (local address 127.15.254.3:43567)
0807 06:25:10.933449 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:10.933518 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:10.936246 (+ 2728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:10.937232 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:10.937244 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:10.937689 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:10.938511 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:10.938536 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:10.939681 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:10.939685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:10.941163 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:10.941176 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:10.941792 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:10.941799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:10.942105 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:10.943181 (+ 1076us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:10.943339 (+ 158us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:10.946574 (+ 3235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:10.949210 (+ 2636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:10.949217 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:10.949237 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:10.949656 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:10.950030 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:10.950033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:10.950035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:10.950238 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:10.950865 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:10.950873 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:10.951238 (+ 365us) client_negotiation.cc:769] Sending connection context
0807 06:25:10.952152 (+ 914us) client_negotiation.cc:240] Negotiation successful
0807 06:25:10.952611 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"thread_start_us":236,"threads_started":1}
I20260807 06:25:10.955227 2005 heartbeater.cc:347] Connected to a master server at 127.15.254.62:35437
I20260807 06:25:10.955519 2005 heartbeater.cc:464] Registering TS with master...
I20260807 06:25:10.956151 2005 heartbeater.cc:511] Master 127.15.254.62:35437 requested a full tablet report, sending...
I20260807 06:25:10.957445 1348 ts_manager.cc:194] Registered new tserver with Master: 68be4dc9ed0b40b4bc2f84e9b35682d8 (127.15.254.3:37133)
I20260807 06:25:10.958261 1348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.3@KRBTEST.COM'} at 127.15.254.3:43567
I20260807 06:25:10.961951 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:25:10.962893 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 1461
I20260807 06:25:10.991952 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 1650
I20260807 06:25:11.020224 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 1833
I20260807 06:25:11.039547 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 1298
I20260807 06:25:11.053753 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:35437
--webserver_interface=127.15.254.62
--webserver_port=33417
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:35203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:35437
--ranger_config_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.254.20:45981/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.254.20:45981/kms with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:25:11.172158 2009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:11.172444 2009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:11.172538 2009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:11.177062 2009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:25:11.177151 2009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:11.177189 2009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:11.177217 2009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:25:11.177263 2009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:25:11.183387 2009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.254.20:45981/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:35437
--ranger_config_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:35437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=33417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:11.184703 2009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:11.185827 2009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:11.195076 2017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:11.195091 2014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:11.195230 2015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:11.195531 2009 server_base.cc:1034] running on GCE node
I20260807 06:25:11.196228 2009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:11.197424 2009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:11.198614 2009 hybrid_clock.cc:648] HybridClock initialized: now 1786083911198555 us; error 78 us; skew 500 ppm
Aug 07 06:25:11 dist-test-slave-44d2 krb5kdc[685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083911, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:11.201722 2009 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:25:11.202682 2009 webserver.cc:460] Webserver started at http://127.15.254.62:33417/ using document root <none> and password file <none>
I20260807 06:25:11.203249 2009 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:11.203321 2009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 06:25:11 dist-test-slave-44d2 krb5kdc[685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083911, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for HTTP/127.15.254.20@KRBTEST.COM
I20260807 06:25:11.258399 2009 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:25:11.261633 2030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:11.263425 2009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:25:11.263582 2009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "e54b0808633548808dd1fcf36d51c7d4"
format_stamp: "Formatted at 2026-08-07 06:25:08 on dist-test-slave-44d2"
server_key: "c8d52bb5ef32e9974af925019e2ec73d"
server_key_iv: "64ae2f78ab316c88f4d9ccdab331d2c8"
server_key_version: "kuduclusterkey@0"
I20260807 06:25:11.264432 2009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:11.284519 2009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:11.288365 2009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:11.288633 2009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:11.298259 2009 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:35437
I20260807 06:25:11.298369 2100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:35437 every 8 connection(s)
I20260807 06:25:11.299897 2009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:25:11.302620 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 2009
I20260807 06:25:11.303221 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.254.20:45981/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:37419
--local_ip_for_outbound_sockets=127.15.254.1
--tserver_master_addrs=127.15.254.62:35437
--webserver_port=45917
--webserver_interface=127.15.254.1
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:35203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.254.20:45981/kms with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:25:11.305711 2101 sys_catalog.cc:263] Verifying existing consensus state
I20260807 06:25:11.307575 2101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4: Bootstrap starting.
I20260807 06:25:11.333534 2101 log.cc:824] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4: Log is configured to *not* fsync() on all Append() calls
I20260807 06:25:11.343079 2101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 06:25:11.343631 2101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4: Bootstrap complete.
I20260807 06:25:11.349830 2101 raft_consensus.cc:348] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54b0808633548808dd1fcf36d51c7d4" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 35437 } }
I20260807 06:25:11.350369 2101 raft_consensus.cc:729] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e54b0808633548808dd1fcf36d51c7d4, State: Initialized, Role: FOLLOWER
I20260807 06:25:11.350981 2101 consensus_queue.cc:260] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54b0808633548808dd1fcf36d51c7d4" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 35437 } }
I20260807 06:25:11.351151 2101 raft_consensus.cc:388] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:25:11.351229 2101 raft_consensus.cc:482] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:25:11.351353 2101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [term 1 FOLLOWER]: Advancing to term 2
I20260807 06:25:11.354218 2101 raft_consensus.cc:504] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54b0808633548808dd1fcf36d51c7d4" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 35437 } }
I20260807 06:25:11.354617 2101 leader_election.cc:302] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e54b0808633548808dd1fcf36d51c7d4; no voters:
I20260807 06:25:11.355051 2101 leader_election.cc:288] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 06:25:11.355319 2105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 06:25:11.355660 2105 raft_consensus.cc:686] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [term 2 LEADER]: Becoming Leader. State: Replica: e54b0808633548808dd1fcf36d51c7d4, State: Running, Role: LEADER
I20260807 06:25:11.356074 2105 consensus_queue.cc:237] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54b0808633548808dd1fcf36d51c7d4" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 35437 } }
I20260807 06:25:11.356525 2101 sys_catalog.cc:564] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:25:11.357323 2105 sys_catalog.cc:455] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e54b0808633548808dd1fcf36d51c7d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54b0808633548808dd1fcf36d51c7d4" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 35437 } } }
I20260807 06:25:11.357549 2105 sys_catalog.cc:458] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [sys.catalog]: This master's current role is: LEADER
I20260807 06:25:11.357940 2105 sys_catalog.cc:455] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e54b0808633548808dd1fcf36d51c7d4. Latest consensus state: current_term: 2 leader_uuid: "e54b0808633548808dd1fcf36d51c7d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54b0808633548808dd1fcf36d51c7d4" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 35437 } } }
I20260807 06:25:11.358090 2105 sys_catalog.cc:458] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4 [sys.catalog]: This master's current role is: LEADER
I20260807 06:25:11.358413 2117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:25:11.362062 2117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:25:11.362679 2101 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 06:25:11.362874 2117 catalog_manager.cc:1259] Loaded cluster ID: 6059d08ee70c445980b7bad1dd9c5bc2
I20260807 06:25:11.362972 2117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:25:11.365356 2117 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:25:11.366087 2117 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4: Loaded TSK: 0
I20260807 06:25:11.367427 2117 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 06:25:11.474598 2103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:11.474997 2103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:11.475080 2103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:11.481592 2103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:11.481702 2103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:11.481806 2103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:25:11.489691 2103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.254.20:45981/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:37419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=45917
--tserver_master_addrs=127.15.254.62:35437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:11.491098 2103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:11.492525 2103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:11.506471 2158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:11.506426 2157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:11.507014 2103 server_base.cc:1034] running on GCE node
W20260807 06:25:11.507272 2161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:11.507771 2103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:11.508682 2103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:11.509908 2103 hybrid_clock.cc:648] HybridClock initialized: now 1786083911509874 us; error 58 us; skew 500 ppm
Aug 07 06:25:11 dist-test-slave-44d2 krb5kdc[685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083911, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:11.514115 2103 init.cc:377] Logged in from keytab as kudu/127.15.254.1@KRBTEST.COM (short username kudu)
I20260807 06:25:11.515427 2103 webserver.cc:460] Webserver started at http://127.15.254.1:45917/ using document root <none> and password file <none>
I20260807 06:25:11.516245 2103 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:11.516448 2103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 06:25:11 dist-test-slave-44d2 krb5kdc[685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083911, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for HTTP/127.15.254.20@KRBTEST.COM
I20260807 06:25:11.549089 2103 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.003s sys 0.001s
I20260807 06:25:11.551961 2175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:11.553562 2103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:25:11.553692 2103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "e688f4fbe7244ee6a61c824db42dd62d"
format_stamp: "Formatted at 2026-08-07 06:25:09 on dist-test-slave-44d2"
server_key: "2158e5d383228c9e0a2e7e5e9cb9e252"
server_key_iv: "61b9360ecb491e8f8b5d35539c22a05e"
server_key_version: "kuduclusterkey@0"
I20260807 06:25:11.554549 2103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:11.579763 2103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:11.583458 2103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:11.583665 2103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:11.584363 2103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:25:11.585500 2103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:25:11.585563 2103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:11.585623 2103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:25:11.585649 2103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:11.598326 2103 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:37419
I20260807 06:25:11.599731 2103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:25:11.606848 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 2103
I20260807 06:25:11.607378 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.254.20:45981/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:34607
--local_ip_for_outbound_sockets=127.15.254.2
--tserver_master_addrs=127.15.254.62:35437
--webserver_port=33421
--webserver_interface=127.15.254.2
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:35203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.254.20:45981/kms with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:25:11.602841 2340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:37419 every 8 connection(s)
Aug 07 06:25:11 dist-test-slave-44d2 krb5kdc[685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083911, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:25:11.640718 2347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:11.609779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:59955 (local address 127.15.254.62:35437)
0807 06:25:11.611125 (+ 1346us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:11.611150 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:11.613563 (+ 2413us) server_negotiation.cc:407] Connection header received
0807 06:25:11.614518 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:11.614536 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:11.615006 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:11.615412 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:11.617322 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:11.618930 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:11.620740 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:11.621525 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:11.621704 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:11.629863 (+ 8159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:11.629879 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:11.629900 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:11.629940 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:11.632128 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:11.633585 (+ 1457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:11.633592 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:11.633598 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:11.633760 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:11.634250 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:11.634254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:11.634255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:11.634808 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:11.634962 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:11.639119 (+ 4157us) server_negotiation.cc:299] Negotiation successful
0807 06:25:11.639505 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1155,"thread_start_us":126,"threads_started":1}
I20260807 06:25:11.642475 2345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:11.605247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:35437 (local address 127.15.254.1:59955)
0807 06:25:11.612814 (+ 7567us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:11.612880 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:11.613943 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:11.615990 (+ 2047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:11.615999 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:11.616421 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:11.617109 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:11.617129 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:11.619103 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:11.619106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:11.620586 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:11.620599 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:11.621954 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:11.621958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:11.622168 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:11.623555 (+ 1387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:11.623580 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:11.628633 (+ 5053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:11.632280 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:11.632286 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:11.632303 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:11.632745 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:11.633912 (+ 1167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:11.633916 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:11.633918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:11.634118 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:11.637288 (+ 3170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:11.637295 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:11.638686 (+ 1391us) client_negotiation.cc:769] Sending connection context
0807 06:25:11.641123 (+ 2437us) client_negotiation.cc:240] Negotiation successful
0807 06:25:11.641485 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7320,"thread_start_us":162,"threads_started":1}
I20260807 06:25:11.646469 2341 heartbeater.cc:347] Connected to a master server at 127.15.254.62:35437
I20260807 06:25:11.647384 2341 heartbeater.cc:464] Registering TS with master...
I20260807 06:25:11.648064 2341 heartbeater.cc:511] Master 127.15.254.62:35437 requested a full tablet report, sending...
W20260807 06:25:11.657555 2341 heartbeater.cc:644] Failed to heartbeat to 127.15.254.62:35437 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 06:25:11.657730 2341 heartbeater.cc:464] Registering TS with master...
I20260807 06:25:11.657975 2341 heartbeater.cc:511] Master 127.15.254.62:35437 requested a full tablet report, sending...
I20260807 06:25:11.659022 2341 heartbeater.cc:464] Registering TS with master...
I20260807 06:25:11.659209 2341 heartbeater.cc:511] Master 127.15.254.62:35437 requested a full tablet report, sending...
W20260807 06:25:11.660034 2341 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 06:25:11.795106 2346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:11.795418 2346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:11.795542 2346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:11.800140 2346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:11.800235 2346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:11.800328 2346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:25:11.806434 2346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.254.20:45981/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:34607
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=33421
--tserver_master_addrs=127.15.254.62:35437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:11.807682 2346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:11.808828 2346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:11.819972 2355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:11.819976 2359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:11.820098 2354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:11.820753 2346 server_base.cc:1034] running on GCE node
I20260807 06:25:11.821319 2346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:11.822244 2346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:11.823462 2346 hybrid_clock.cc:648] HybridClock initialized: now 1786083911823418 us; error 58 us; skew 500 ppm
Aug 07 06:25:11 dist-test-slave-44d2 krb5kdc[685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083911, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:11.827358 2346 init.cc:377] Logged in from keytab as kudu/127.15.254.2@KRBTEST.COM (short username kudu)
I20260807 06:25:11.828594 2346 webserver.cc:460] Webserver started at http://127.15.254.2:33421/ using document root <none> and password file <none>
I20260807 06:25:11.829387 2346 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:11.829496 2346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 06:25:11 dist-test-slave-44d2 krb5kdc[685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083911, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for HTTP/127.15.254.20@KRBTEST.COM
I20260807 06:25:11.858044 2346 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.001s
I20260807 06:25:11.861837 2371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:11.863672 2346 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 06:25:11.863837 2346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "420fe814e5da47cda4cc863ba8fd3401"
format_stamp: "Formatted at 2026-08-07 06:25:10 on dist-test-slave-44d2"
server_key: "69e31988843f53c2f0beb1061bc8cf6a"
server_key_iv: "07eaf7d99946a3986cee1343ab16859a"
server_key_version: "kuduclusterkey@0"
I20260807 06:25:11.864732 2346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:11.898048 2346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:11.902114 2346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:11.902359 2346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:11.903185 2346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:25:11.904387 2346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:25:11.904459 2346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:11.904564 2346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:25:11.904606 2346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:11.916436 2346 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:34607
I20260807 06:25:11.916628 2527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:34607 every 8 connection(s)
I20260807 06:25:11.917768 2346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:25:11.927044 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 2346
I20260807 06:25:11.927608 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.254.20:45981/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:37133
--local_ip_for_outbound_sockets=127.15.254.3
--tserver_master_addrs=127.15.254.62:35437
--webserver_port=32801
--webserver_interface=127.15.254.3
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:35203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.254.20:45981/kms with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 06:25:11 dist-test-slave-44d2 krb5kdc[685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083911, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:25:11.946648 2347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:11.922469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:34181 (local address 127.15.254.62:35437)
0807 06:25:11.922619 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:11.922625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:11.923027 (+ 402us) server_negotiation.cc:407] Connection header received
0807 06:25:11.924422 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:11.924426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:11.924519 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:11.924584 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:11.925994 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:11.926548 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:11.928427 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:11.928889 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:11.928992 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:11.933537 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:11.933542 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:11.933545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:11.933578 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:11.935487 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:11.939701 (+ 4214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:11.939705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:11.939707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:11.939810 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:11.945083 (+ 5273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:11.945088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:11.945090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:11.945373 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:11.945470 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:11.946297 (+ 827us) server_negotiation.cc:299] Negotiation successful
0807 06:25:11.946464 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260807 06:25:11.948418 2531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:11.921233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:35437 (local address 127.15.254.2:34181)
0807 06:25:11.922833 (+ 1600us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:11.922906 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:11.924142 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:11.924997 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:11.925004 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:11.925302 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:11.925812 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:11.925825 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:11.926744 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:11.926749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:11.928232 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:11.928248 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:11.929237 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:11.929243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:11.929508 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:11.930302 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:11.930330 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:11.933287 (+ 2957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:11.936281 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:11.936288 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:11.936308 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:11.939558 (+ 3250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:11.939947 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:11.939951 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:11.939953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:11.940167 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:11.945556 (+ 5389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:11.945564 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:11.945969 (+ 405us) client_negotiation.cc:769] Sending connection context
0807 06:25:11.946872 (+ 903us) client_negotiation.cc:240] Negotiation successful
0807 06:25:11.947282 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1371,"thread_start_us":163,"threads_started":1}
I20260807 06:25:11.949990 2529 heartbeater.cc:347] Connected to a master server at 127.15.254.62:35437
I20260807 06:25:11.950286 2529 heartbeater.cc:464] Registering TS with master...
I20260807 06:25:11.950986 2529 heartbeater.cc:511] Master 127.15.254.62:35437 requested a full tablet report, sending...
W20260807 06:25:11.952962 2529 heartbeater.cc:644] Failed to heartbeat to 127.15.254.62:35437 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 06:25:11.953112 2529 heartbeater.cc:464] Registering TS with master...
I20260807 06:25:11.953325 2529 heartbeater.cc:511] Master 127.15.254.62:35437 requested a full tablet report, sending...
I20260807 06:25:11.954351 2529 heartbeater.cc:464] Registering TS with master...
I20260807 06:25:11.954548 2529 heartbeater.cc:511] Master 127.15.254.62:35437 requested a full tablet report, sending...
W20260807 06:25:11.955406 2529 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 06:25:12.095970 2533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:12.096321 2533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:12.096405 2533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:12.101200 2533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:12.101293 2533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:12.101508 2533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:25:12.109319 2533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.254.20:45981/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:37133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=32801
--tserver_master_addrs=127.15.254.62:35437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:12.110532 2533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:12.111634 2533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:12.122292 2540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:12.122629 2533 server_base.cc:1034] running on GCE node
W20260807 06:25:12.122872 2545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:12.123462 2541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:12.124046 2533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:12.124975 2533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:12.126186 2533 hybrid_clock.cc:648] HybridClock initialized: now 1786083912126160 us; error 44 us; skew 500 ppm
Aug 07 06:25:12 dist-test-slave-44d2 krb5kdc[685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083912, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:12.130038 2533 init.cc:377] Logged in from keytab as kudu/127.15.254.3@KRBTEST.COM (short username kudu)
I20260807 06:25:12.131217 2533 webserver.cc:460] Webserver started at http://127.15.254.3:32801/ using document root <none> and password file <none>
I20260807 06:25:12.131956 2533 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:12.132067 2533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 06:25:12 dist-test-slave-44d2 krb5kdc[685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083912, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for HTTP/127.15.254.20@KRBTEST.COM
I20260807 06:25:12.159214 2533 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 06:25:12.162590 2553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:12.164116 2533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 06:25:12.164285 2533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "68be4dc9ed0b40b4bc2f84e9b35682d8"
format_stamp: "Formatted at 2026-08-07 06:25:10 on dist-test-slave-44d2"
server_key: "b51ab230a7aa16c19d229b33a1a83e68"
server_key_iv: "bd22178549a0590bb6fa5d7d42ee4042"
server_key_version: "kuduclusterkey@0"
I20260807 06:25:12.165094 2533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:12.201418 2533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:12.205451 2533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:12.205690 2533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:12.206427 2533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:25:12.207528 2533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:25:12.207592 2533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:12.207648 2533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:25:12.207674 2533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:12.221733 2533 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:37133
I20260807 06:25:12.223304 2533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:25:12.223270 2717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:37133 every 8 connection(s)
I20260807 06:25:12.228858 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 2533
Aug 07 06:25:12 dist-test-slave-44d2 krb5kdc[685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083912, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:25:12.242132 2347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:12.227411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:59281 (local address 127.15.254.62:35437)
0807 06:25:12.227578 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:12.227583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:12.227602 (+ 19us) server_negotiation.cc:407] Connection header received
0807 06:25:12.228053 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:12.228056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:12.228113 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:12.228200 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:12.229799 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:12.230374 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:12.231999 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:12.232343 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:12.232429 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:12.235700 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:12.235704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:12.235707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:12.235737 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:12.237701 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:12.240103 (+ 2402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:12.240107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:12.240109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:12.240204 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:12.240689 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:12.240693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:12.240695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:12.240929 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:12.241007 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:12.241795 (+ 788us) server_negotiation.cc:299] Negotiation successful
0807 06:25:12.241943 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 06:25:12.244144 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:12.226209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:35437 (local address 127.15.254.3:59281)
0807 06:25:12.226759 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:12.226815 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:12.227828 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:12.228414 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:12.228424 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:12.228831 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:12.229605 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:12.229628 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:12.230515 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:12.230518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:12.231826 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:12.231839 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:12.232711 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:12.232715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:12.232932 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:12.233739 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:12.233761 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:12.235485 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:12.239513 (+ 4028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:12.239520 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:12.239539 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:12.239969 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:12.240338 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:12.240341 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:12.240343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:12.240563 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:12.241104 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:12.241111 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:12.241480 (+ 369us) client_negotiation.cc:769] Sending connection context
0807 06:25:12.242623 (+ 1143us) client_negotiation.cc:240] Negotiation successful
0807 06:25:12.243014 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":161,"threads_started":1}
I20260807 06:25:12.245819 2718 heartbeater.cc:347] Connected to a master server at 127.15.254.62:35437
I20260807 06:25:12.246155 2718 heartbeater.cc:464] Registering TS with master...
I20260807 06:25:12.246717 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:12.233773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53938 (local address 127.15.254.62:35437)
0807 06:25:12.236319 (+ 2546us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:12.236332 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:12.236855 (+ 523us) server_negotiation.cc:407] Connection header received
0807 06:25:12.237113 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:12.237116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:12.237167 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:12.237230 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:12.238303 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:12.239165 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:12.240449 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:12.240831 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:12.240922 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:12.242701 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:12.242705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:12.242708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:12.242733 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:12.244562 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:12.245102 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:12.245106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:12.245107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:12.245211 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:12.245573 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:12.245577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:12.245578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:12.245794 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:12.246010 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:12.246408 (+ 398us) server_negotiation.cc:299] Negotiation successful
0807 06:25:12.246536 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2442,"thread_start_us":108,"threads_started":1}
I20260807 06:25:12.247165 2718 heartbeater.cc:511] Master 127.15.254.62:35437 requested a full tablet report, sending...
W20260807 06:25:12.249450 2718 heartbeater.cc:644] Failed to heartbeat to 127.15.254.62:35437 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 06:25:12.249655 2718 heartbeater.cc:464] Registering TS with master...
I20260807 06:25:12.249917 2718 heartbeater.cc:511] Master 127.15.254.62:35437 requested a full tablet report, sending...
I20260807 06:25:12.250967 2718 heartbeater.cc:464] Registering TS with master...
I20260807 06:25:12.251211 2718 heartbeater.cc:511] Master 127.15.254.62:35437 requested a full tablet report, sending...
W20260807 06:25:12.252189 2718 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 06:25:12.660307 2341 heartbeater.cc:464] Registering TS with master...
I20260807 06:25:12.660727 2341 heartbeater.cc:511] Master 127.15.254.62:35437 requested a full tablet report, sending...
I20260807 06:25:12.662094 2053 ts_manager.cc:194] Registered new tserver with Master: e688f4fbe7244ee6a61c824db42dd62d (127.15.254.1:37419)
I20260807 06:25:12.663909 2053 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.1@KRBTEST.COM'} at 127.15.254.1:59955
Aug 07 06:25:12 dist-test-slave-44d2 krb5kdc[685](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786083912, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:12.955670 2529 heartbeater.cc:464] Registering TS with master...
I20260807 06:25:12.956112 2529 heartbeater.cc:511] Master 127.15.254.62:35437 requested a full tablet report, sending...
I20260807 06:25:12.957203 2053 ts_manager.cc:194] Registered new tserver with Master: 420fe814e5da47cda4cc863ba8fd3401 (127.15.254.2:34607)
I20260807 06:25:12.958097 2053 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.2@KRBTEST.COM'} at 127.15.254.2:34181
Aug 07 06:25:13 dist-test-slave-44d2 krb5kdc[685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083912, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for HTTP/127.15.254.20@KRBTEST.COM
I20260807 06:25:13.252547 2718 heartbeater.cc:464] Registering TS with master...
I20260807 06:25:13.252945 2718 heartbeater.cc:511] Master 127.15.254.62:35437 requested a full tablet report, sending...
I20260807 06:25:13.253846 2053 ts_manager.cc:194] Registered new tserver with Master: 68be4dc9ed0b40b4bc2f84e9b35682d8 (127.15.254.3:37133)
I20260807 06:25:13.254618 2053 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.3@KRBTEST.COM'} at 127.15.254.3:59281
I20260807 06:25:13.263119 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:25:13.297907 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:13.269053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53954 (local address 127.15.254.62:35437)
0807 06:25:13.276628 (+ 7575us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:13.276633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:13.276652 (+ 19us) server_negotiation.cc:407] Connection header received
0807 06:25:13.276706 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:13.276709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:13.276762 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:13.276887 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:13.283905 (+ 7018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:13.285341 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:13.287147 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:13.290971 (+ 3824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:13.291098 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:13.293438 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:13.293445 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:13.293452 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:13.293499 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:13.295636 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:13.296175 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:13.296179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:13.296181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:13.296280 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:13.296625 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:13.296629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:13.296631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:13.296893 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:13.297032 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:13.297500 (+ 468us) server_negotiation.cc:299] Negotiation successful
0807 06:25:13.297654 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7393,"thread_start_us":89,"threads_started":1}
I20260807 06:25:13.310578 2053 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:25:13.315724 2053 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:25:13.358613 2759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:13.341384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:34607 (local address 127.0.0.1:47486)
0807 06:25:13.348973 (+ 7589us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:13.348983 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:13.349084 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:13.350917 (+ 1833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:13.350922 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:13.350945 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:25:13.351072 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:13.351081 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:13.352963 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:13.352970 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:13.355859 (+ 2889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:13.355875 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:13.357666 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:13.357672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:13.357892 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:13.357976 (+ 84us) client_negotiation.cc:769] Sending connection context
0807 06:25:13.358104 (+ 128us) client_negotiation.cc:240] Negotiation successful
0807 06:25:13.358341 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7546,"thread_start_us":133,"threads_started":1}
I20260807 06:25:13.359372 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:13.342111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47486 (local address 127.15.254.2:34607)
0807 06:25:13.342415 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:13.342423 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:13.349089 (+ 6666us) server_negotiation.cc:407] Connection header received
0807 06:25:13.349200 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:13.349208 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:13.350472 (+ 1264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:13.350648 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:25:13.351226 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:13.352802 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:13.356058 (+ 3256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:13.357487 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:13.358864 (+ 1377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:13.359057 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:13.359121 (+ 64us) server_negotiation.cc:299] Negotiation successful
0807 06:25:13.359231 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":101,"threads_started":1}
I20260807 06:25:13.359467 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:13.340130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:37133 (local address 127.0.0.1:33492)
0807 06:25:13.341783 (+ 1653us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:13.341841 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:13.341989 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:13.350233 (+ 8244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:13.350241 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:13.350270 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:25:13.350458 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:13.350467 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:13.356334 (+ 5867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:13.356338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:13.357646 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:13.357660 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:13.359030 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:13.359034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:13.359132 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:13.359160 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 06:25:13.359216 (+ 56us) client_negotiation.cc:240] Negotiation successful
0807 06:25:13.359293 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1458,"thread_start_us":118,"threads_started":1}
I20260807 06:25:13.360296 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:13.340293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33492 (local address 127.15.254.3:37133)
0807 06:25:13.348751 (+ 8458us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:13.348759 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:13.348790 (+ 31us) server_negotiation.cc:407] Connection header received
0807 06:25:13.348888 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:13.348895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:13.349156 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:13.350647 (+ 1491us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:25:13.350833 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:13.352087 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:13.357902 (+ 5815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:13.358809 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:13.359779 (+ 970us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:13.359971 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:13.360032 (+ 61us) server_negotiation.cc:299] Negotiation successful
0807 06:25:13.360142 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8316,"thread_start_us":86,"threads_started":1}
I20260807 06:25:13.363663 2619 tablet_service.cc:1467] Processing CreateTablet for tablet 76a6a895822e41ca9a1b64051dfd5538 (DEFAULT_TABLE table=test-table [id=369ca3a8cf84444a8e57531957543d34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:25:13.363737 2431 tablet_service.cc:1467] Processing CreateTablet for tablet 76a6a895822e41ca9a1b64051dfd5538 (DEFAULT_TABLE table=test-table [id=369ca3a8cf84444a8e57531957543d34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:25:13.365092 2431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a6a895822e41ca9a1b64051dfd5538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:25:13.365142 2619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a6a895822e41ca9a1b64051dfd5538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:25:13.380388 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:13.340411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40760 (local address 127.15.254.1:37419)
0807 06:25:13.349505 (+ 9094us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:13.349512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:13.349535 (+ 23us) server_negotiation.cc:407] Connection header received
0807 06:25:13.349608 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:13.349613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:13.349798 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:13.349963 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:25:13.369227 (+ 19264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:13.370775 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:13.377921 (+ 7146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:13.379111 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:13.379241 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:13.379430 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:13.379986 (+ 556us) server_negotiation.cc:299] Negotiation successful
0807 06:25:13.380147 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8976,"thread_start_us":126,"threads_started":1}
I20260807 06:25:13.381314 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:13.340231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:37419 (local address 127.0.0.1:40760)
0807 06:25:13.348720 (+ 8489us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:13.348734 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:13.348834 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:13.359279 (+ 10445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:13.359283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:13.359311 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:25:13.359412 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:13.359422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:13.371194 (+ 11772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:13.371200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:13.372794 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:13.372813 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:13.379579 (+ 6766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:13.379584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:13.379762 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:13.379802 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 06:25:13.380954 (+ 1152us) client_negotiation.cc:240] Negotiation successful
0807 06:25:13.381089 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8418,"thread_start_us":83,"threads_started":1}
I20260807 06:25:13.384967 2767 tablet_bootstrap.cc:492] T 76a6a895822e41ca9a1b64051dfd5538 P 68be4dc9ed0b40b4bc2f84e9b35682d8: Bootstrap starting.
I20260807 06:25:13.385344 2238 tablet_service.cc:1467] Processing CreateTablet for tablet 76a6a895822e41ca9a1b64051dfd5538 (DEFAULT_TABLE table=test-table [id=369ca3a8cf84444a8e57531957543d34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:25:13.386626 2238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a6a895822e41ca9a1b64051dfd5538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:25:13.387699 2767 tablet_bootstrap.cc:654] T 76a6a895822e41ca9a1b64051dfd5538 P 68be4dc9ed0b40b4bc2f84e9b35682d8: Neither blocks nor log segments found. Creating new log.
I20260807 06:25:13.388695 2767 log.cc:824] T 76a6a895822e41ca9a1b64051dfd5538 P 68be4dc9ed0b40b4bc2f84e9b35682d8: Log is configured to *not* fsync() on all Append() calls
I20260807 06:25:13.391553 2767 tablet_bootstrap.cc:492] T 76a6a895822e41ca9a1b64051dfd5538 P 68be4dc9ed0b40b4bc2f84e9b35682d8: No bootstrap required, opened a new log
I20260807 06:25:13.391798 2767 ts_tablet_manager.cc:1397] T 76a6a895822e41ca9a1b64051dfd5538 P 68be4dc9ed0b40b4bc2f84e9b35682d8: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20260807 06:25:13.396955 2767 raft_consensus.cc:348] T 76a6a895822e41ca9a1b64051dfd5538 P 68be4dc9ed0b40b4bc2f84e9b35682d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68be4dc9ed0b40b4bc2f84e9b35682d8" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 37133 } } peers { permanent_uuid: "e688f4fbe7244ee6a61c824db42dd62d" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 37419 } } peers { permanent_uuid: "420fe814e5da47cda4cc863ba8fd3401" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34607 } }
I20260807 06:25:13.397276 2767 raft_consensus.cc:374] T 76a6a895822e41ca9a1b64051dfd5538 P 68be4dc9ed0b40b4bc2f84e9b35682d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:25:13.397339 2767 raft_consensus.cc:729] T 76a6a895822e41ca9a1b64051dfd5538 P 68be4dc9ed0b40b4bc2f84e9b35682d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68be4dc9ed0b40b4bc2f84e9b35682d8, State: Initialized, Role: FOLLOWER
I20260807 06:25:13.397925 2767 consensus_queue.cc:260] T 76a6a895822e41ca9a1b64051dfd5538 P 68be4dc9ed0b40b4bc2f84e9b35682d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68be4dc9ed0b40b4bc2f84e9b35682d8" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 37133 } } peers { permanent_uuid: "e688f4fbe7244ee6a61c824db42dd62d" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 37419 } } peers { permanent_uuid: "420fe814e5da47cda4cc863ba8fd3401" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34607 } }
I20260807 06:25:13.399194 2767 ts_tablet_manager.cc:1428] T 76a6a895822e41ca9a1b64051dfd5538 P 68be4dc9ed0b40b4bc2f84e9b35682d8: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260807 06:25:13.399628 2718 heartbeater.cc:503] Master 127.15.254.62:35437 was elected leader, sending a full tablet report...
I20260807 06:25:13.408958 2773 tablet_bootstrap.cc:492] T 76a6a895822e41ca9a1b64051dfd5538 P e688f4fbe7244ee6a61c824db42dd62d: Bootstrap starting.
I20260807 06:25:13.411892 2773 tablet_bootstrap.cc:654] T 76a6a895822e41ca9a1b64051dfd5538 P e688f4fbe7244ee6a61c824db42dd62d: Neither blocks nor log segments found. Creating new log.
I20260807 06:25:13.413036 2773 log.cc:824] T 76a6a895822e41ca9a1b64051dfd5538 P e688f4fbe7244ee6a61c824db42dd62d: Log is configured to *not* fsync() on all Append() calls
I20260807 06:25:13.413951 2769 tablet_bootstrap.cc:492] T 76a6a895822e41ca9a1b64051dfd5538 P 420fe814e5da47cda4cc863ba8fd3401: Bootstrap starting.
I20260807 06:25:13.417768 2769 tablet_bootstrap.cc:654] T 76a6a895822e41ca9a1b64051dfd5538 P 420fe814e5da47cda4cc863ba8fd3401: Neither blocks nor log segments found. Creating new log.
I20260807 06:25:13.419353 2769 log.cc:824] T 76a6a895822e41ca9a1b64051dfd5538 P 420fe814e5da47cda4cc863ba8fd3401: Log is configured to *not* fsync() on all Append() calls
I20260807 06:25:13.422966 2773 tablet_bootstrap.cc:492] T 76a6a895822e41ca9a1b64051dfd5538 P e688f4fbe7244ee6a61c824db42dd62d: No bootstrap required, opened a new log
I20260807 06:25:13.423236 2773 ts_tablet_manager.cc:1397] T 76a6a895822e41ca9a1b64051dfd5538 P e688f4fbe7244ee6a61c824db42dd62d: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20260807 06:25:13.428203 2773 raft_consensus.cc:348] T 76a6a895822e41ca9a1b64051dfd5538 P e688f4fbe7244ee6a61c824db42dd62d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68be4dc9ed0b40b4bc2f84e9b35682d8" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 37133 } } peers { permanent_uuid: "e688f4fbe7244ee6a61c824db42dd62d" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 37419 } } peers { permanent_uuid: "420fe814e5da47cda4cc863ba8fd3401" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34607 } }
I20260807 06:25:13.428601 2773 raft_consensus.cc:374] T 76a6a895822e41ca9a1b64051dfd5538 P e688f4fbe7244ee6a61c824db42dd62d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:25:13.428680 2773 raft_consensus.cc:729] T 76a6a895822e41ca9a1b64051dfd5538 P e688f4fbe7244ee6a61c824db42dd62d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e688f4fbe7244ee6a61c824db42dd62d, State: Initialized, Role: FOLLOWER
I20260807 06:25:13.431447 2769 tablet_bootstrap.cc:492] T 76a6a895822e41ca9a1b64051dfd5538 P 420fe814e5da47cda4cc863ba8fd3401: No bootstrap required, opened a new log
I20260807 06:25:13.431710 2769 ts_tablet_manager.cc:1397] T 76a6a895822e41ca9a1b64051dfd5538 P 420fe814e5da47cda4cc863ba8fd3401: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20260807 06:25:13.438086 2769 raft_consensus.cc:348] T 76a6a895822e41ca9a1b64051dfd5538 P 420fe814e5da47cda4cc863ba8fd3401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68be4dc9ed0b40b4bc2f84e9b35682d8" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 37133 } } peers { permanent_uuid: "e688f4fbe7244ee6a61c824db42dd62d" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 37419 } } peers { permanent_uuid: "420fe814e5da47cda4cc863ba8fd3401" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34607 } }
I20260807 06:25:13.438491 2769 raft_consensus.cc:374] T 76a6a895822e41ca9a1b64051dfd5538 P 420fe814e5da47cda4cc863ba8fd3401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:25:13.438572 2769 raft_consensus.cc:729] T 76a6a895822e41ca9a1b64051dfd5538 P 420fe814e5da47cda4cc863ba8fd3401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 420fe814e5da47cda4cc863ba8fd3401, State: Initialized, Role: FOLLOWER
I20260807 06:25:13.439275 2773 consensus_queue.cc:260] T 76a6a895822e41ca9a1b64051dfd5538 P e688f4fbe7244ee6a61c824db42dd62d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68be4dc9ed0b40b4bc2f84e9b35682d8" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 37133 } } peers { permanent_uuid: "e688f4fbe7244ee6a61c824db42dd62d" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 37419 } } peers { permanent_uuid: "420fe814e5da47cda4cc863ba8fd3401" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34607 } }
I20260807 06:25:13.440742 2773 ts_tablet_manager.cc:1428] T 76a6a895822e41ca9a1b64051dfd5538 P e688f4fbe7244ee6a61c824db42dd62d: Time spent starting tablet: real 0.017s user 0.010s sys 0.000s
I20260807 06:25:13.441314 2769 consensus_queue.cc:260] T 76a6a895822e41ca9a1b64051dfd5538 P 420fe814e5da47cda4cc863ba8fd3401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68be4dc9ed0b40b4bc2f84e9b35682d8" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 37133 } } peers { permanent_uuid: "e688f4fbe7244ee6a61c824db42dd62d" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 37419 } } peers { permanent_uuid: "420fe814e5da47cda4cc863ba8fd3401" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34607 } }
I20260807 06:25:13.442484 2769 ts_tablet_manager.cc:1428] T 76a6a895822e41ca9a1b64051dfd5538 P 420fe814e5da47cda4cc863ba8fd3401: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260807 06:25:13.442899 2341 heartbeater.cc:503] Master 127.15.254.62:35437 was elected leader, sending a full tablet report...
I20260807 06:25:13.446353 2529 heartbeater.cc:503] Master 127.15.254.62:35437 was elected leader, sending a full tablet report...
W20260807 06:25:13.475575 2719 tablet.cc:2369] T 76a6a895822e41ca9a1b64051dfd5538 P 68be4dc9ed0b40b4bc2f84e9b35682d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:25:13.481626 2777 raft_consensus.cc:482] T 76a6a895822e41ca9a1b64051dfd5538 P e688f4fbe7244ee6a61c824db42dd62d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:25:13.481899 2777 raft_consensus.cc:504] T 76a6a895822e41ca9a1b64051dfd5538 P e688f4fbe7244ee6a61c824db42dd62d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68be4dc9ed0b40b4bc2f84e9b35682d8" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 37133 } } peers { permanent_uuid: "e688f4fbe7244ee6a61c824db42dd62d" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 37419 } } peers { permanent_uuid: "420fe814e5da47cda4cc863ba8fd3401" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34607 } }
I20260807 06:25:13.483505 2777 leader_election.cc:288] T 76a6a895822e41ca9a1b64051dfd5538 P e688f4fbe7244ee6a61c824db42dd62d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68be4dc9ed0b40b4bc2f84e9b35682d8 (127.15.254.3:37133), 420fe814e5da47cda4cc863ba8fd3401 (127.15.254.2:34607)
I20260807 06:25:13.493017 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:13.484670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:48807 (local address 127.15.254.3:37133)
0807 06:25:13.484798 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:13.484803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:13.486800 (+ 1997us) server_negotiation.cc:407] Connection header received
0807 06:25:13.487064 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:13.487069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:13.487126 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:13.487179 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:25:13.487720 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:13.488533 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:13.491481 (+ 2948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:13.492153 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:13.492260 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:13.492316 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:13.492802 (+ 486us) server_negotiation.cc:299] Negotiation successful
0807 06:25:13.492884 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260807 06:25:13.493033 2780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:13.483704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:37133 (local address 127.15.254.1:48807)
0807 06:25:13.486619 (+ 2915us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:13.486636 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:13.486914 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:13.487333 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:13.487336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:13.487355 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:25:13.487561 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:13.487569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:13.489908 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:13.489914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:13.491340 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:13.491349 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:13.492425 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:13.492429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:13.492644 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:13.492680 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 06:25:13.492773 (+ 93us) client_negotiation.cc:240] Negotiation successful
0807 06:25:13.492865 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2836,"thread_start_us":183,"threads_started":1}
I20260807 06:25:13.493772 2634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a6a895822e41ca9a1b64051dfd5538" candidate_uuid: "e688f4fbe7244ee6a61c824db42dd62d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68be4dc9ed0b40b4bc2f84e9b35682d8" is_pre_election: true
I20260807 06:25:13.494185 2634 raft_consensus.cc:2454] T 76a6a895822e41ca9a1b64051dfd5538 P 68be4dc9ed0b40b4bc2f84e9b35682d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e688f4fbe7244ee6a61c824db42dd62d in term 0.
I20260807 06:25:13.494841 2178 leader_election.cc:302] T 76a6a895822e41ca9a1b64051dfd5538 P e688f4fbe7244ee6a61c824db42dd62d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68be4dc9ed0b40b4bc2f84e9b35682d8, e688f4fbe7244ee6a61c824db42dd62d; no voters:
I20260807 06:25:13.495220 2777 raft_consensus.cc:2781] T 76a6a895822e41ca9a1b64051dfd5538 P e688f4fbe7244ee6a61c824db42dd62d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:25:13.495321 2777 raft_consensus.cc:482] T 76a6a895822e41ca9a1b64051dfd5538 P e688f4fbe7244ee6a61c824db42dd62d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:25:13.495373 2777 raft_consensus.cc:3037] T 76a6a895822e41ca9a1b64051dfd5538 P e688f4fbe7244ee6a61c824db42dd62d [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:25:13.497074 2777 raft_consensus.cc:504] T 76a6a895822e41ca9a1b64051dfd5538 P e688f4fbe7244ee6a61c824db42dd62d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68be4dc9ed0b40b4bc2f84e9b35682d8" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 37133 } } peers { permanent_uuid: "e688f4fbe7244ee6a61c824db42dd62d" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 37419 } } peers { permanent_uuid: "420fe814e5da47cda4cc863ba8fd3401" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34607 } }
I20260807 06:25:13.497504 2777 leader_election.cc:288] T 76a6a895822e41ca9a1b64051dfd5538 P e688f4fbe7244ee6a61c824db42dd62d [CANDIDATE]: Term 1 election: Requested vote from peers 68be4dc9ed0b40b4bc2f84e9b35682d8 (127.15.254.3:37133), 420fe814e5da47cda4cc863ba8fd3401 (127.15.254.2:34607)
I20260807 06:25:13.498035 2634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a6a895822e41ca9a1b64051dfd5538" candidate_uuid: "e688f4fbe7244ee6a61c824db42dd62d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68be4dc9ed0b40b4bc2f84e9b35682d8"
I20260807 06:25:13.498176 2634 raft_consensus.cc:3037] T 76a6a895822e41ca9a1b64051dfd5538 P 68be4dc9ed0b40b4bc2f84e9b35682d8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:25:13.499586 2634 raft_consensus.cc:2454] T 76a6a895822e41ca9a1b64051dfd5538 P 68be4dc9ed0b40b4bc2f84e9b35682d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e688f4fbe7244ee6a61c824db42dd62d in term 1.
I20260807 06:25:13.499955 2178 leader_election.cc:302] T 76a6a895822e41ca9a1b64051dfd5538 P e688f4fbe7244ee6a61c824db42dd62d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68be4dc9ed0b40b4bc2f84e9b35682d8, e688f4fbe7244ee6a61c824db42dd62d; no voters:
I20260807 06:25:13.500137 2777 raft_consensus.cc:2781] T 76a6a895822e41ca9a1b64051dfd5538 P e688f4fbe7244ee6a61c824db42dd62d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:25:13.500376 2777 raft_consensus.cc:686] T 76a6a895822e41ca9a1b64051dfd5538 P e688f4fbe7244ee6a61c824db42dd62d [term 1 LEADER]: Becoming Leader. State: Replica: e688f4fbe7244ee6a61c824db42dd62d, State: Running, Role: LEADER
I20260807 06:25:13.500836 2777 consensus_queue.cc:237] T 76a6a895822e41ca9a1b64051dfd5538 P e688f4fbe7244ee6a61c824db42dd62d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68be4dc9ed0b40b4bc2f84e9b35682d8" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 37133 } } peers { permanent_uuid: "e688f4fbe7244ee6a61c824db42dd62d" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 37419 } } peers { permanent_uuid: "420fe814e5da47cda4cc863ba8fd3401" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34607 } }
Aug 07 06:25:13 dist-test-slave-44d2 krb5kdc[685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083912, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for HTTP/127.15.254.20@KRBTEST.COM
I20260807 06:25:13.505357 2053 catalog_manager.cc:5495] T 76a6a895822e41ca9a1b64051dfd5538 P e688f4fbe7244ee6a61c824db42dd62d reported cstate change: term changed from 0 to 1, leader changed from <none> to e688f4fbe7244ee6a61c824db42dd62d (127.15.254.1). New cstate: current_term: 1 leader_uuid: "e688f4fbe7244ee6a61c824db42dd62d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68be4dc9ed0b40b4bc2f84e9b35682d8" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 37133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e688f4fbe7244ee6a61c824db42dd62d" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 37419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "420fe814e5da47cda4cc863ba8fd3401" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34607 } health_report { overall_health: UNKNOWN } } }
I20260807 06:25:13.512791 2781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:13.484216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:34607 (local address 127.15.254.1:54637)
0807 06:25:13.490938 (+ 6722us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:13.490953 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:13.491191 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:13.504589 (+ 13398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:13.504594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:13.504621 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:25:13.504834 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:13.504845 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:13.509635 (+ 4790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:13.509639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:13.510882 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:13.510898 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:13.512127 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:13.512132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:13.512274 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:13.512315 (+ 41us) client_negotiation.cc:769] Sending connection context
0807 06:25:13.512384 (+ 69us) client_negotiation.cc:240] Negotiation successful
0807 06:25:13.512520 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6644,"thread_start_us":94,"threads_started":1}
I20260807 06:25:13.513439 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:13.484350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:54637 (local address 127.15.254.2:34607)
0807 06:25:13.488291 (+ 3941us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:13.488296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:13.491164 (+ 2868us) server_negotiation.cc:407] Connection header received
0807 06:25:13.491286 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:13.491290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:13.491347 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:13.491470 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:25:13.505345 (+ 13875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:13.506154 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:13.511047 (+ 4893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:13.511794 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:13.511964 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:13.512048 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:13.513154 (+ 1106us) server_negotiation.cc:299] Negotiation successful
0807 06:25:13.513275 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3838}
I20260807 06:25:13.514206 2458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a6a895822e41ca9a1b64051dfd5538" candidate_uuid: "e688f4fbe7244ee6a61c824db42dd62d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "420fe814e5da47cda4cc863ba8fd3401"
I20260807 06:25:13.514484 2458 raft_consensus.cc:3037] T 76a6a895822e41ca9a1b64051dfd5538 P 420fe814e5da47cda4cc863ba8fd3401 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:25:13.516405 2458 raft_consensus.cc:2454] T 76a6a895822e41ca9a1b64051dfd5538 P 420fe814e5da47cda4cc863ba8fd3401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e688f4fbe7244ee6a61c824db42dd62d in term 1.
I20260807 06:25:13.514179 2467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a6a895822e41ca9a1b64051dfd5538" candidate_uuid: "e688f4fbe7244ee6a61c824db42dd62d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "420fe814e5da47cda4cc863ba8fd3401" is_pre_election: true
I20260807 06:25:13.516645 2467 raft_consensus.cc:2362] T 76a6a895822e41ca9a1b64051dfd5538 P 420fe814e5da47cda4cc863ba8fd3401 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate e688f4fbe7244ee6a61c824db42dd62d in term 1. Re-sending same reply.
I20260807 06:25:13.538378 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:13.533012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40768 (local address 127.15.254.1:37419)
0807 06:25:13.533165 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:13.533171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:13.533189 (+ 18us) server_negotiation.cc:407] Connection header received
0807 06:25:13.533287 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:13.533291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:13.533374 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:13.533499 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:25:13.533786 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:13.534603 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:13.535917 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:13.536249 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:13.536358 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:13.536886 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:25:13.537521 (+ 635us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:25:13.537656 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:13.538124 (+ 468us) server_negotiation.cc:299] Negotiation successful
0807 06:25:13.538197 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 06:25:13.546650 2634 raft_consensus.cc:1261] T 76a6a895822e41ca9a1b64051dfd5538 P 68be4dc9ed0b40b4bc2f84e9b35682d8 [term 1 FOLLOWER]: Refusing update from remote peer e688f4fbe7244ee6a61c824db42dd62d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:25:13.547544 2777 consensus_queue.cc:1035] T 76a6a895822e41ca9a1b64051dfd5538 P e688f4fbe7244ee6a61c824db42dd62d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68be4dc9ed0b40b4bc2f84e9b35682d8" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 37133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:25:13.547852 2467 raft_consensus.cc:1261] T 76a6a895822e41ca9a1b64051dfd5538 P 420fe814e5da47cda4cc863ba8fd3401 [term 1 FOLLOWER]: Refusing update from remote peer e688f4fbe7244ee6a61c824db42dd62d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:25:13.548583 2777 consensus_queue.cc:1035] T 76a6a895822e41ca9a1b64051dfd5538 P e688f4fbe7244ee6a61c824db42dd62d [LEADER]: Connected to new peer: Peer: permanent_uuid: "420fe814e5da47cda4cc863ba8fd3401" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:25:13.567471 2786 mvcc.cc:205] Tried to move back new op lower bound from 7315799709876199424 to 7315799709706240000. Current Snapshot: MvccSnapshot[applied={T|T < 7315799709876199424}]
I20260807 06:25:13.585253 2053 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:25:13.585464 2053 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:25:13.588352 2053 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e54b0808633548808dd1fcf36d51c7d4: Sending DeleteTablet for 3 replicas of tablet 76a6a895822e41ca9a1b64051dfd5538
I20260807 06:25:13.589308 2619 tablet_service.cc:1514] Processing DeleteTablet for tablet 76a6a895822e41ca9a1b64051dfd5538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:25:13 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:33492
I20260807 06:25:13.589509 2238 tablet_service.cc:1514] Processing DeleteTablet for tablet 76a6a895822e41ca9a1b64051dfd5538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:25:13 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:40760
I20260807 06:25:13.590008 2431 tablet_service.cc:1514] Processing DeleteTablet for tablet 76a6a895822e41ca9a1b64051dfd5538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:25:13 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:47486
I20260807 06:25:13.590723 2804 tablet_replica.cc:330] T 76a6a895822e41ca9a1b64051dfd5538 P 68be4dc9ed0b40b4bc2f84e9b35682d8: stopping tablet replica
I20260807 06:25:13.591145 2804 raft_consensus.cc:2229] T 76a6a895822e41ca9a1b64051dfd5538 P 68be4dc9ed0b40b4bc2f84e9b35682d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:25:13.591478 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 2103
I20260807 06:25:13.590485 2808 tablet_replica.cc:330] T 76a6a895822e41ca9a1b64051dfd5538 P 420fe814e5da47cda4cc863ba8fd3401: stopping tablet replica
I20260807 06:25:13.591953 2808 raft_consensus.cc:2229] T 76a6a895822e41ca9a1b64051dfd5538 P 420fe814e5da47cda4cc863ba8fd3401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:25:13.592435 2804 raft_consensus.cc:2258] T 76a6a895822e41ca9a1b64051dfd5538 P 68be4dc9ed0b40b4bc2f84e9b35682d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:25:13.592338 2808 raft_consensus.cc:2258] T 76a6a895822e41ca9a1b64051dfd5538 P 420fe814e5da47cda4cc863ba8fd3401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:25:13.597177 2804 ts_tablet_manager.cc:1905] T 76a6a895822e41ca9a1b64051dfd5538 P 68be4dc9ed0b40b4bc2f84e9b35682d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:25:13.600852 2804 ts_tablet_manager.cc:1918] T 76a6a895822e41ca9a1b64051dfd5538 P 68be4dc9ed0b40b4bc2f84e9b35682d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:25:13.601037 2804 log.cc:1196] T 76a6a895822e41ca9a1b64051dfd5538 P 68be4dc9ed0b40b4bc2f84e9b35682d8: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/76a6a895822e41ca9a1b64051dfd5538
I20260807 06:25:13.601461 2804 ts_tablet_manager.cc:1939] T 76a6a895822e41ca9a1b64051dfd5538 P 68be4dc9ed0b40b4bc2f84e9b35682d8: Deleting consensus metadata
I20260807 06:25:13.604252 2808 ts_tablet_manager.cc:1905] T 76a6a895822e41ca9a1b64051dfd5538 P 420fe814e5da47cda4cc863ba8fd3401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:25:13.604295 2032 catalog_manager.cc:4841] TS 68be4dc9ed0b40b4bc2f84e9b35682d8 (127.15.254.3:37133): tablet 76a6a895822e41ca9a1b64051dfd5538 (table test-table [id=369ca3a8cf84444a8e57531957543d34]) successfully deleted
I20260807 06:25:13.606913 2808 ts_tablet_manager.cc:1918] T 76a6a895822e41ca9a1b64051dfd5538 P 420fe814e5da47cda4cc863ba8fd3401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:25:13.607049 2808 log.cc:1196] T 76a6a895822e41ca9a1b64051dfd5538 P 420fe814e5da47cda4cc863ba8fd3401: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/76a6a895822e41ca9a1b64051dfd5538
I20260807 06:25:13.607349 2808 ts_tablet_manager.cc:1939] T 76a6a895822e41ca9a1b64051dfd5538 P 420fe814e5da47cda4cc863ba8fd3401: Deleting consensus metadata
I20260807 06:25:13.608630 2032 catalog_manager.cc:4841] TS 420fe814e5da47cda4cc863ba8fd3401 (127.15.254.2:34607): tablet 76a6a895822e41ca9a1b64051dfd5538 (table test-table [id=369ca3a8cf84444a8e57531957543d34]) successfully deleted
W20260807 06:25:13.613159 2033 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.254.1:37419 (error 108)
I20260807 06:25:13.614431 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 2346
I20260807 06:25:13.614780 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:13.614413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:37419 (local address 127.0.0.1:40772)
0807 06:25:13.614612 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:13.614683 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.1:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260807 06:25:13.617138 2033 catalog_manager.cc:4584] TS e688f4fbe7244ee6a61c824db42dd62d (127.15.254.1:37419): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 76a6a895822e41ca9a1b64051dfd5538: Network error: Client connection negotiation failed: client connection to 127.15.254.1:37419: connect: Connection refused (error 111)
I20260807 06:25:13.629750 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 2533
I20260807 06:25:13.646155 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 2009
I20260807 06:25:13.666006 16376 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260807 06:25:14.021787 16376 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260807 06:25:14.021968 16376 mini_ranger.cc:66] Stopping Ranger...
I20260807 06:25:14.388054 16376 mini_ranger.cc:68] Stopped Ranger
2026-08-07 06:25:14.388 UTC [721] LOG: received smart shutdown request
2026-08-07 06:25:14.393 UTC [721] LOG: background worker "logical replication launcher" (PID 734) exited with exit code 1
2026-08-07 06:25:14.393 UTC [729] LOG: shutting down
2026-08-07 06:25:14.598 UTC [721] LOG: database system is shut down
2026-08-07T06:25:14Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55536 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:25:14 dist-test-slave-44d2 krb5kdc[2825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:25:14 dist-test-slave-44d2 krb5kdc[2825](info): set up 2 sockets
Aug 07 06:25:14 dist-test-slave-44d2 krb5kdc[2825](info): commencing operation
krb5kdc: starting...
W20260807 06:25:15.962189 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:25:16 dist-test-slave-44d2 krb5kdc[2825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:25:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:25:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:25:16.103343 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:37999
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:37999,127.15.254.61:43345,127.15.254.60:46753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:25:16.219095 2843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:16.219411 2843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:16.219489 2843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:16.223521 2843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:25:16.223613 2843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:16.223654 2843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:16.223708 2843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:25:16.223742 2843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:25:16.229202 2843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:37999,127.15.254.61:43345,127.15.254.60:46753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:37999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:16.230317 2843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:16.231381 2843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:16.240319 2854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:16.240325 2852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:16.240561 2843 server_base.cc:1034] running on GCE node
W20260807 06:25:16.240319 2850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:16.241321 2843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:16.242462 2843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:16.243692 2843 hybrid_clock.cc:648] HybridClock initialized: now 1786083916243637 us; error 57 us; skew 500 ppm
Aug 07 06:25:16 dist-test-slave-44d2 krb5kdc[2825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083916, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:16.246654 2843 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:25:16.247706 2843 webserver.cc:460] Webserver started at http://127.15.254.62:41733/ using document root <none> and password file <none>
I20260807 06:25:16.248303 2843 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:16.248394 2843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:25:16.248668 2843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:25:16.250799 2843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "d62684b49ee54dfc83bd7cf7da4fc516"
format_stamp: "Formatted at 2026-08-07 06:25:16 on dist-test-slave-44d2"
server_key: "2c9476bd328a1966e7712384faa66eb0"
server_key_iv: "70cfc913b8f23d78e7547549f4201441"
server_key_version: "encryptionkey@0"
I20260807 06:25:16.251389 2843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "d62684b49ee54dfc83bd7cf7da4fc516"
format_stamp: "Formatted at 2026-08-07 06:25:16 on dist-test-slave-44d2"
server_key: "2c9476bd328a1966e7712384faa66eb0"
server_key_iv: "70cfc913b8f23d78e7547549f4201441"
server_key_version: "encryptionkey@0"
I20260807 06:25:16.255810 2843 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260807 06:25:16.258805 2862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:16.260388 2843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 06:25:16.260576 2843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "d62684b49ee54dfc83bd7cf7da4fc516"
format_stamp: "Formatted at 2026-08-07 06:25:16 on dist-test-slave-44d2"
server_key: "2c9476bd328a1966e7712384faa66eb0"
server_key_iv: "70cfc913b8f23d78e7547549f4201441"
server_key_version: "encryptionkey@0"
I20260807 06:25:16.260689 2843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:16.274927 2843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:16.278644 2843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:16.278888 2843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:16.286763 2843 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:37999
I20260807 06:25:16.286792 2937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:37999 every 8 connection(s)
I20260807 06:25:16.287838 2843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:25:16.291096 2939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:25:16.291321 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 2843
I20260807 06:25:16.291448 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:25:16.291765 16376 external_mini_cluster.cc:1428] Setting key 06be5c9718a0334ccd5b09aed08c449a
I20260807 06:25:16.295595 2939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37999 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.254.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.61@KRBTEST.COM" created.
Aug 07 06:25:16 dist-test-slave-44d2 krb5kdc[2825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083916, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:25:16.312713 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.298266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:37999 (local address 127.0.0.1:45102)
0807 06:25:16.298826 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.298890 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:16.299829 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:16.300764 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:16.300769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:16.301143 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:16.301512 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:16.301530 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.302776 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.302779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:16.303609 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:16.303649 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.304332 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.304338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:16.304514 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:16.305201 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:16.305219 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:16.306746 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:16.309241 (+ 2495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:16.309250 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:16.309259 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:16.309514 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:16.310023 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:16.310026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:16.310028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:16.310151 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:16.310831 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:16.310838 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:16.310997 (+ 159us) client_negotiation.cc:769] Sending connection context
0807 06:25:16.311182 (+ 185us) client_negotiation.cc:240] Negotiation successful
0807 06:25:16.311623 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":132,"thread_start_us":164,"threads_started":1}
I20260807 06:25:16.312713 2944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.298267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45102 (local address 127.15.254.62:37999)
0807 06:25:16.298923 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:16.298937 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:16.298988 (+ 51us) server_negotiation.cc:407] Connection header received
0807 06:25:16.300194 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:16.300201 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:16.300473 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:16.300744 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:16.301684 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.302634 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.303760 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.304137 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.304259 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:16.306928 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:16.306934 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:16.306945 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:16.306968 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:16.309104 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.309642 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.309652 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.309657 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.309834 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.310271 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.310277 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.310280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.310709 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:16.310880 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:16.311252 (+ 372us) server_negotiation.cc:299] Negotiation successful
0807 06:25:16.311540 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"spinlock_wait_cycles":9088,"thread_start_us":194,"threads_started":1}
I20260807 06:25:16.315248 2939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.61" port: 43345 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:25:16.316221 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.315768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:43345 (local address 127.0.0.1:43302)
0807 06:25:16.316048 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.316150 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.61:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260807 06:25:16.316423 2868 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.254.61:43345: connect: Connection refused (error 111)
I20260807 06:25:16.317412 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.317192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:43345 (local address 127.0.0.1:43306)
0807 06:25:16.317305 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.317357 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.61:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260807 06:25:16.317596 2939 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.61:43345: Network error: Client connection negotiation failed: client connection to 127.15.254.61:43345: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.61.keytab.
Entry for principal kudu/127.15.254.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.61.keytab.
I20260807 06:25:16.324128 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.61:43345
--webserver_interface=127.15.254.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.61.keytab
--principal=kudu/127.15.254.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:37999,127.15.254.61:43345,127.15.254.60:46753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:25:16.366865 2939 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.61" port: 43345 } attempt: 1
I20260807 06:25:16.367751 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.367446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:43345 (local address 127.0.0.1:43312)
0807 06:25:16.367599 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.367671 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.61:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260807 06:25:16.368636 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.368348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:43345 (local address 127.0.0.1:43326)
0807 06:25:16.368501 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.368560 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.61:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260807 06:25:16.368794 2939 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.61:43345: Network error: Client connection negotiation failed: client connection to 127.15.254.61:43345: connect: Connection refused (error 111)
I20260807 06:25:16.436980 2939 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.61" port: 43345 } attempt: 2
I20260807 06:25:16.437845 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.437537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:43345 (local address 127.0.0.1:43334)
0807 06:25:16.437704 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.437769 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.61:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260807 06:25:16.438674 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.438420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:43345 (local address 127.0.0.1:43344)
0807 06:25:16.438558 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.438608 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.61:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260807 06:25:16.438886 2939 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.61:43345: Network error: Client connection negotiation failed: client connection to 127.15.254.61:43345: connect: Connection refused (error 111)
W20260807 06:25:16.444614 2947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:16.444868 2947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:16.444938 2947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:16.448944 2947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:25:16.449002 2947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:16.449024 2947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:16.449043 2947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:25:16.449100 2947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:25:16.454591 2947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:37999,127.15.254.61:43345,127.15.254.60:46753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.61.keytab
--principal=kudu/127.15.254.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.61:43345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.254.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:16.455703 2947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:16.456800 2947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:16.466200 2954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:16.466329 2952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:16.466576 2947 server_base.cc:1034] running on GCE node
W20260807 06:25:16.466711 2957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:16.467398 2947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:16.468755 2947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:16.469959 2947 hybrid_clock.cc:648] HybridClock initialized: now 1786083916469937 us; error 41 us; skew 500 ppm
Aug 07 06:25:16 dist-test-slave-44d2 krb5kdc[2825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083916, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:16.473222 2947 init.cc:377] Logged in from keytab as kudu/127.15.254.61@KRBTEST.COM (short username kudu)
I20260807 06:25:16.474272 2947 webserver.cc:460] Webserver started at http://127.15.254.61:34069/ using document root <none> and password file <none>
I20260807 06:25:16.474886 2947 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:16.474990 2947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:25:16.475243 2947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:25:16.477464 2947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-1/data/instance:
uuid: "6103ac5e06ff48d185e311207c64c017"
format_stamp: "Formatted at 2026-08-07 06:25:16 on dist-test-slave-44d2"
server_key: "278148e7f6cc43aab399bcb57b66bc4c"
server_key_iv: "bf410b6264e47d083e92a3d3f815af07"
server_key_version: "encryptionkey@0"
I20260807 06:25:16.478076 2947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-1/wal/instance:
uuid: "6103ac5e06ff48d185e311207c64c017"
format_stamp: "Formatted at 2026-08-07 06:25:16 on dist-test-slave-44d2"
server_key: "278148e7f6cc43aab399bcb57b66bc4c"
server_key_iv: "bf410b6264e47d083e92a3d3f815af07"
server_key_version: "encryptionkey@0"
I20260807 06:25:16.482206 2947 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260807 06:25:16.485008 2966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:16.486400 2947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 06:25:16.486558 2947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-1/wal
uuid: "6103ac5e06ff48d185e311207c64c017"
format_stamp: "Formatted at 2026-08-07 06:25:16 on dist-test-slave-44d2"
server_key: "278148e7f6cc43aab399bcb57b66bc4c"
server_key_iv: "bf410b6264e47d083e92a3d3f815af07"
server_key_version: "encryptionkey@0"
I20260807 06:25:16.486757 2947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:16.504113 2947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:16.507293 2947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:16.507476 2947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:16.515219 2947 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.61:43345
I20260807 06:25:16.515249 3044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.61:43345 every 8 connection(s)
I20260807 06:25:16.516186 2947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-1/data/info.pb
I20260807 06:25:16.519157 3045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:25:16.522548 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 2947
I20260807 06:25:16.522665 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-1/wal/instance
I20260807 06:25:16.522991 16376 external_mini_cluster.cc:1428] Setting key 0dab62cddce6698099b3969f514c9666
I20260807 06:25:16.523218 3045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37999 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:25:16.530079 2939 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.61" port: 43345 } attempt: 3
WARNING: no policy specified for kudu/127.15.254.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.60@KRBTEST.COM" created.
Aug 07 06:25:16 dist-test-slave-44d2 krb5kdc[2825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083916, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.61@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
Aug 07 06:25:16 dist-test-slave-44d2 krb5kdc[2825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083916, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for kudu/127.15.254.61@KRBTEST.COM
I20260807 06:25:16.541931 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.530550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:43345 (local address 127.0.0.1:43360)
0807 06:25:16.530742 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.530756 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:16.530833 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:16.531607 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:16.531610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:16.531815 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:16.531948 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:16.531957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.533418 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.533421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:16.534003 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:16.534012 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.534611 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.534613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:16.534694 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:16.535199 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:16.535211 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:16.537542 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:16.539876 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:16.539879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:16.539881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:16.540149 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:16.540536 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:16.540539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:16.540540 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:16.540643 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:16.541251 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:16.541257 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:16.541394 (+ 137us) client_negotiation.cc:769] Sending connection context
0807 06:25:16.541510 (+ 116us) client_negotiation.cc:240] Negotiation successful
0807 06:25:16.541696 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"spinlock_wait_cycles":11648}
I20260807 06:25:16.542680 2944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.525140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45108 (local address 127.15.254.62:37999)
0807 06:25:16.525331 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:16.525339 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:16.526234 (+ 895us) server_negotiation.cc:407] Connection header received
0807 06:25:16.527181 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:16.527188 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:16.527264 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:16.527374 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:16.528757 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.529687 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.531443 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.531860 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.531990 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:16.537055 (+ 5065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:16.537060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:16.537067 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:16.537091 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:16.538917 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.539519 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.539524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.539526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.539631 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.540031 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.540037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.540039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.540283 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:16.540403 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:16.541896 (+ 1493us) server_negotiation.cc:299] Negotiation successful
0807 06:25:16.542532 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 06:25:16.542639 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.525548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:37999 (local address 127.0.0.1:45108)
0807 06:25:16.526055 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.526112 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:16.526959 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:16.527544 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:16.527551 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:16.528000 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:16.528591 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:16.528609 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.529850 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.529853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:16.531288 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:16.531296 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.532042 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.532047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:16.532257 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:16.532998 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:16.533019 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:16.536901 (+ 3882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:16.539046 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:16.539055 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:16.539067 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:16.539401 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:16.539741 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:16.539745 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:16.539747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:16.539943 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:16.540403 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:16.540411 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:16.540839 (+ 428us) client_negotiation.cc:769] Sending connection context
0807 06:25:16.541052 (+ 213us) client_negotiation.cc:240] Negotiation successful
0807 06:25:16.541469 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"spinlock_wait_cycles":768,"thread_start_us":155,"threads_started":1}
I20260807 06:25:16.542799 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.530652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43360 (local address 127.15.254.61:43345)
0807 06:25:16.531112 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:16.531123 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:16.531148 (+ 25us) server_negotiation.cc:407] Connection header received
0807 06:25:16.531189 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:16.531194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:16.531448 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:16.531616 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:16.532088 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.533279 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.534150 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.534508 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.534595 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:16.537723 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:16.537734 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:16.537751 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:16.537784 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:16.539755 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.540266 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.540276 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.540280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.540375 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.540746 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.540752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.540757 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.541114 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:16.542198 (+ 1084us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:16.542316 (+ 118us) server_negotiation.cc:299] Negotiation successful
0807 06:25:16.542474 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"spinlock_wait_cycles":4992,"thread_start_us":172,"threads_started":1}
I20260807 06:25:16.544317 3045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.61" port: 43345 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:25:16.545347 2939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:25:16.546110 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.545847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46753 (local address 127.0.0.1:35444)
0807 06:25:16.545986 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.546043 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260807 06:25:16.546875 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.546629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46753 (local address 127.0.0.1:35458)
0807 06:25:16.546765 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.546812 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260807 06:25:16.547051 2939 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.60:46753: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46753: connect: Connection refused (error 111)
Aug 07 06:25:16 dist-test-slave-44d2 krb5kdc[2825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083916, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.61@KRBTEST.COM for kudu/127.15.254.61@KRBTEST.COM
I20260807 06:25:16.553124 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.544939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43374 (local address 127.15.254.61:43345)
0807 06:25:16.545066 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:16.545072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:16.545090 (+ 18us) server_negotiation.cc:407] Connection header received
0807 06:25:16.545135 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:16.545139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:16.545189 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:16.545279 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:16.545731 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.546424 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.547443 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.547740 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.547853 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:16.550052 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:16.550056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:16.550059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:16.550082 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:16.551391 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.551820 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.551825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.551826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.551910 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.552209 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.552213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.552214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.552394 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:16.552519 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:16.552820 (+ 301us) server_negotiation.cc:299] Negotiation successful
0807 06:25:16.552994 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":50}
I20260807 06:25:16.553124 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.544883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:43345 (local address 127.0.0.1:43374)
0807 06:25:16.544974 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.544991 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:16.545070 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:16.545309 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:16.545312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:16.545486 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:16.545608 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:16.545615 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.546550 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.546554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:16.547328 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:16.547338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.547852 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.547855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:16.547958 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:16.548393 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:16.548411 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:16.549920 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:16.551495 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:16.551500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:16.551502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:16.551725 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:16.552005 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:16.552008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:16.552010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:16.552090 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:16.552544 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:16.552547 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:16.552639 (+ 92us) client_negotiation.cc:769] Sending connection context
0807 06:25:16.552772 (+ 133us) client_negotiation.cc:240] Negotiation successful
0807 06:25:16.552944 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260807 06:25:16.554245 3045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:25:16.554940 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.554695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46753 (local address 127.0.0.1:35472)
0807 06:25:16.554797 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.554873 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.60.keytab.
Entry for principal kudu/127.15.254.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.60.keytab.
W20260807 06:25:16.555619 2969 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46753: connect: Connection refused (error 111)
I20260807 06:25:16.556404 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.60:46753
--webserver_interface=127.15.254.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.60.keytab
--principal=kudu/127.15.254.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:37999,127.15.254.61:43345,127.15.254.60:46753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:25:16.556589 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.556352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46753 (local address 127.0.0.1:35480)
0807 06:25:16.556459 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.556528 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260807 06:25:16.556737 3045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.60:46753: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46753: connect: Connection refused (error 111)
I20260807 06:25:16.578222 2939 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } attempt: 1
I20260807 06:25:16.579187 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.578831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46753 (local address 127.0.0.1:35492)
0807 06:25:16.579031 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.579104 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260807 06:25:16.580108 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.579812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46753 (local address 127.0.0.1:35506)
0807 06:25:16.580000 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.580048 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260807 06:25:16.580253 2939 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.60:46753: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46753: connect: Connection refused (error 111)
I20260807 06:25:16.606062 3045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } attempt: 1
I20260807 06:25:16.607103 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.606770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46753 (local address 127.0.0.1:35516)
0807 06:25:16.606950 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.607017 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260807 06:25:16.607863 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.607652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46753 (local address 127.0.0.1:35518)
0807 06:25:16.607760 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.607811 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260807 06:25:16.608067 3045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.60:46753: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46753: connect: Connection refused (error 111)
I20260807 06:25:16.655436 2939 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } attempt: 2
I20260807 06:25:16.656329 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.655994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46753 (local address 127.0.0.1:35524)
0807 06:25:16.656144 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.656211 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260807 06:25:16.657035 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.656817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46753 (local address 127.0.0.1:35536)
0807 06:25:16.656923 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.656971 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260807 06:25:16.657158 2939 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.60:46753: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46753: connect: Connection refused (error 111)
I20260807 06:25:16.676234 3045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } attempt: 2
I20260807 06:25:16.677189 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.676809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46753 (local address 127.0.0.1:35544)
0807 06:25:16.677029 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.677098 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260807 06:25:16.678114 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.677819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46753 (local address 127.0.0.1:35552)
0807 06:25:16.677999 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.678054 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260807 06:25:16.678256 3045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.60:46753: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46753: connect: Connection refused (error 111)
W20260807 06:25:16.683817 3054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:16.684104 3054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:16.684199 3054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:16.688199 3054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:25:16.688278 3054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:16.688323 3054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:16.688380 3054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:25:16.688416 3054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:25:16.693866 3054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:37999,127.15.254.61:43345,127.15.254.60:46753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.60.keytab
--principal=kudu/127.15.254.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.60:46753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.254.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:16.695015 3054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:16.696137 3054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:16.705507 3059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:16.705507 3060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:16.705610 3062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:16.706187 3054 server_base.cc:1034] running on GCE node
I20260807 06:25:16.706806 3054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:16.708122 3054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:16.709331 3054 hybrid_clock.cc:648] HybridClock initialized: now 1786083916709312 us; error 38 us; skew 500 ppm
Aug 07 06:25:16 dist-test-slave-44d2 krb5kdc[2825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083916, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:16.712616 3054 init.cc:377] Logged in from keytab as kudu/127.15.254.60@KRBTEST.COM (short username kudu)
I20260807 06:25:16.713685 3054 webserver.cc:460] Webserver started at http://127.15.254.60:37225/ using document root <none> and password file <none>
I20260807 06:25:16.714279 3054 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:16.714376 3054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:25:16.714602 3054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:25:16.716804 3054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-2/data/instance:
uuid: "190c4aeb89cc48ce95ef424f465abd26"
format_stamp: "Formatted at 2026-08-07 06:25:16 on dist-test-slave-44d2"
server_key: "9e13fc4298a86364675582c8db4e47e6"
server_key_iv: "a1cdf119f9b5feb2053316d6238c5ae2"
server_key_version: "encryptionkey@0"
I20260807 06:25:16.717429 3054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-2/wal/instance:
uuid: "190c4aeb89cc48ce95ef424f465abd26"
format_stamp: "Formatted at 2026-08-07 06:25:16 on dist-test-slave-44d2"
server_key: "9e13fc4298a86364675582c8db4e47e6"
server_key_iv: "a1cdf119f9b5feb2053316d6238c5ae2"
server_key_version: "encryptionkey@0"
I20260807 06:25:16.721446 3054 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 06:25:16.724080 3072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:16.725435 3054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 06:25:16.725589 3054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-2/wal
uuid: "190c4aeb89cc48ce95ef424f465abd26"
format_stamp: "Formatted at 2026-08-07 06:25:16 on dist-test-slave-44d2"
server_key: "9e13fc4298a86364675582c8db4e47e6"
server_key_iv: "a1cdf119f9b5feb2053316d6238c5ae2"
server_key_version: "encryptionkey@0"
I20260807 06:25:16.725706 3054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:16.754069 3054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:16.756337 2939 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } attempt: 3
I20260807 06:25:16.757194 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.756901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46753 (local address 127.0.0.1:35564)
0807 06:25:16.757043 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.757111 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260807 06:25:16.758041 3054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:16.758183 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.757875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46753 (local address 127.0.0.1:35580)
0807 06:25:16.758006 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.758064 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260807 06:25:16.758323 2939 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.60:46753: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46753: connect: Connection refused (error 111)
I20260807 06:25:16.758301 3054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:16.766489 3054 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.60:46753
I20260807 06:25:16.766520 3143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.60:46753 every 8 connection(s)
I20260807 06:25:16.767580 3054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-2/data/info.pb
I20260807 06:25:16.769420 3045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } attempt: 3
I20260807 06:25:16.770989 3145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:25:16.774914 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 3054
I20260807 06:25:16.775049 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786083738959718-16376-0/minicluster-data/master-2/wal/instance
I20260807 06:25:16.775368 16376 external_mini_cluster.cc:1428] Setting key b439d668b282494e4d7fa8e2f1646dcc
I20260807 06:25:16.775071 3145 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37999 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:25:16 dist-test-slave-44d2 krb5kdc[2825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083916, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.61@KRBTEST.COM for kudu/127.15.254.60@KRBTEST.COM
Aug 07 06:25:16 dist-test-slave-44d2 krb5kdc[2825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.60@KRBTEST.COM
Aug 07 06:25:16 dist-test-slave-44d2 krb5kdc[2825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083916, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.60@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:25:16.784021 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.769970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46753 (local address 127.0.0.1:35582)
0807 06:25:16.770125 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.770143 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:16.770245 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:16.772695 (+ 2450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:16.772698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:16.772915 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:16.773035 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:16.773043 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.774378 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.774386 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:16.775137 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:16.775151 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.776167 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.776171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:16.776265 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:16.777382 (+ 1117us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:16.777398 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:16.779016 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:16.781605 (+ 2589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:16.781611 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:16.781615 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:16.781920 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:16.782390 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:16.782394 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:16.782396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:16.782512 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:16.783323 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:16.783328 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:16.783450 (+ 122us) client_negotiation.cc:769] Sending connection context
0807 06:25:16.783577 (+ 127us) client_negotiation.cc:240] Negotiation successful
0807 06:25:16.783752 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260807 06:25:16.785485 3148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.770762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35582 (local address 127.15.254.60:46753)
0807 06:25:16.771258 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:16.771273 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:16.771309 (+ 36us) server_negotiation.cc:407] Connection header received
0807 06:25:16.772004 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:16.772022 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:16.772349 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:16.772763 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:16.773221 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.774231 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.775292 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.775995 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.776590 (+ 595us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:16.779198 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:16.779203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:16.779212 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:16.779241 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:16.781461 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.782089 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.782099 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.782109 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.782278 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.782619 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.782623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.782625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.783180 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:16.783343 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:16.784062 (+ 719us) server_negotiation.cc:299] Negotiation successful
0807 06:25:16.784388 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":153,"threads_started":1}
I20260807 06:25:16.788281 3151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.776039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35598 (local address 127.15.254.60:46753)
0807 06:25:16.776871 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:16.776878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:16.776890 (+ 12us) server_negotiation.cc:407] Connection header received
0807 06:25:16.776919 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:16.776922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:16.776957 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:16.777060 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:16.777927 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.778864 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.780047 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.780394 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.780530 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:16.783424 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:16.783428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:16.783430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:16.783454 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:16.786131 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.786647 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.786650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.786652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.786778 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.787088 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.787092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.787094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.787272 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:16.787455 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:16.788032 (+ 577us) server_negotiation.cc:299] Negotiation successful
0807 06:25:16.788129 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":161,"threads_started":1}
I20260807 06:25:16.788761 2944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.777240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45114 (local address 127.15.254.62:37999)
0807 06:25:16.777418 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:16.777425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:16.778265 (+ 840us) server_negotiation.cc:407] Connection header received
0807 06:25:16.778420 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:16.778423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:16.778474 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:16.778569 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:16.779403 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.780140 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.781617 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.781843 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.781959 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:16.784720 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:16.784726 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:16.784729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:16.784761 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:16.786512 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.787063 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.787066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.787068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.787157 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.787482 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.787486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.787487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.787734 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:16.787815 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:16.788448 (+ 633us) server_negotiation.cc:299] Negotiation successful
0807 06:25:16.788615 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 06:25:16.789287 3152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.777194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:37999 (local address 127.0.0.1:45114)
0807 06:25:16.778132 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.778174 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:16.778320 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:16.778626 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:16.778633 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:16.779101 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:16.779255 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:16.779264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.780285 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.780287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:16.781470 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:16.781478 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.781963 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.781967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:16.782091 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:16.782636 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:16.782650 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:16.784576 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:16.786620 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:16.786630 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:16.786636 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:16.786937 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:16.787270 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:16.787273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:16.787274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:16.787373 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:16.788040 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:16.788046 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:16.788172 (+ 126us) client_negotiation.cc:769] Sending connection context
0807 06:25:16.788912 (+ 740us) client_negotiation.cc:240] Negotiation successful
0807 06:25:16.789102 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"thread_start_us":99,"threads_started":1}
I20260807 06:25:16.790485 3145 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.61" port: 43345 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:25:16.790652 3045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017: Bootstrap starting.
I20260807 06:25:16.793612 3045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017: Neither blocks nor log segments found. Creating new log.
I20260807 06:25:16.794603 3045 log.cc:824] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017: Log is configured to *not* fsync() on all Append() calls
Aug 07 06:25:16 dist-test-slave-44d2 krb5kdc[2825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083916, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.60@KRBTEST.COM for kudu/127.15.254.61@KRBTEST.COM
I20260807 06:25:16.797130 3045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017: No bootstrap required, opened a new log
Aug 07 06:25:16 dist-test-slave-44d2 krb5kdc[2825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:25:16.801688 3152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.791057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:43345 (local address 127.0.0.1:43382)
0807 06:25:16.791169 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.791184 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:16.791261 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:16.791484 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:16.791487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:16.791655 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:16.791770 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:16.791777 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.792550 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.792553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:16.793308 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:16.793320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.794394 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.794398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:16.794515 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:16.795712 (+ 1197us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:16.795748 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:16.797692 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:16.799828 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:16.799831 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:16.799833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:16.800083 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:16.800426 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:16.800429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:16.800431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:16.800571 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:16.801149 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:16.801153 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:16.801258 (+ 105us) client_negotiation.cc:769] Sending connection context
0807 06:25:16.801383 (+ 125us) client_negotiation.cc:240] Negotiation successful
0807 06:25:16.801527 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260807 06:25:16.801839 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.791133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.15.254.61:43345)
0807 06:25:16.791248 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:16.791251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:16.791263 (+ 12us) server_negotiation.cc:407] Connection header received
0807 06:25:16.791344 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:16.791346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:16.791382 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:16.791446 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:16.791885 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.792403 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.793471 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.793793 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.793884 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:16.797861 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:16.797865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:16.797868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:16.797895 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:16.799693 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.800199 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.800203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.800205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.800292 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.800689 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.800697 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.800701 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.801022 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:16.801152 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:16.801482 (+ 330us) server_negotiation.cc:299] Negotiation successful
0807 06:25:16.801620 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 06:25:16.801780 3045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62684b49ee54dfc83bd7cf7da4fc516" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37999 } } peers { permanent_uuid: "6103ac5e06ff48d185e311207c64c017" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 43345 } } peers { permanent_uuid: "190c4aeb89cc48ce95ef424f465abd26" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } }
I20260807 06:25:16.802038 3045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:25:16.802091 3045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6103ac5e06ff48d185e311207c64c017, State: Initialized, Role: FOLLOWER
I20260807 06:25:16.802634 3145 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:25:16.802640 3045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62684b49ee54dfc83bd7cf7da4fc516" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37999 } } peers { permanent_uuid: "6103ac5e06ff48d185e311207c64c017" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 43345 } } peers { permanent_uuid: "190c4aeb89cc48ce95ef424f465abd26" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } }
I20260807 06:25:16.804134 2944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.793808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45126 (local address 127.15.254.62:37999)
0807 06:25:16.794053 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:16.794057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:16.794069 (+ 12us) server_negotiation.cc:407] Connection header received
0807 06:25:16.794105 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:16.794108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:16.794148 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:16.794238 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:16.795028 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.795520 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.796419 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.796740 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.796846 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:16.799848 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:16.799852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:16.799860 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:16.799888 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:16.801821 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.802746 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.802750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.802752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.802846 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.803153 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.803157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.803159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.803433 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:16.803684 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:16.803828 (+ 144us) server_negotiation.cc:299] Negotiation successful
0807 06:25:16.803962 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260807 06:25:16.804188 3045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:25:16.807607 3154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62684b49ee54dfc83bd7cf7da4fc516" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37999 } } peers { permanent_uuid: "6103ac5e06ff48d185e311207c64c017" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 43345 } } peers { permanent_uuid: "190c4aeb89cc48ce95ef424f465abd26" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } } }
I20260807 06:25:16.807740 3154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [sys.catalog]: This master's current role is: FOLLOWER
Aug 07 06:25:16 dist-test-slave-44d2 krb5kdc[2825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083916, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.60@KRBTEST.COM for kudu/127.15.254.60@KRBTEST.COM
W20260807 06:25:16.811076 3173 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:25:16.811204 3173 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 07 06:25:16 dist-test-slave-44d2 krb5kdc[2825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.61@KRBTEST.COM
I20260807 06:25:16.808583 2947 master_runner.cc:186] Error getting master registration for 127.15.254.62:37999: OK, instance_id { permanent_uuid: "d62684b49ee54dfc83bd7cf7da4fc516" instance_seqno: 1786083916284537 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 06:25:16.813268 2947 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 06:25:16.815431 3152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.803107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46753 (local address 127.0.0.1:35614)
0807 06:25:16.803318 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.803330 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:16.803413 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:16.803768 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:16.803771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:16.803974 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:16.804069 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:16.804076 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.805142 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.805145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:16.805830 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:16.805840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.806616 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.806621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:16.806732 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:16.809401 (+ 2669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:16.809417 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:16.811104 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:16.813417 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:16.813420 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:16.813422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:16.813627 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:16.814186 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:16.814189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:16.814191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:16.814299 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:16.814841 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:16.814843 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:16.814910 (+ 67us) client_negotiation.cc:769] Sending connection context
0807 06:25:16.815029 (+ 119us) client_negotiation.cc:240] Negotiation successful
0807 06:25:16.815213 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260807 06:25:16.815431 3151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.803339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35614 (local address 127.15.254.60:46753)
0807 06:25:16.803465 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:16.803472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:16.803489 (+ 17us) server_negotiation.cc:407] Connection header received
0807 06:25:16.803528 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:16.803531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:16.803576 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:16.803654 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:16.804207 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.805009 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.805967 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.806345 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.806482 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:16.811283 (+ 4801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:16.811292 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:16.811296 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:16.811329 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:16.813283 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.813979 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.813982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.813984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.814067 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.814430 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.814434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.814437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.814722 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:16.814791 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:16.815084 (+ 293us) server_negotiation.cc:299] Negotiation successful
0807 06:25:16.815283 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":55}
I20260807 06:25:16.817265 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.805898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43386 (local address 127.15.254.61:43345)
0807 06:25:16.806022 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:16.806027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:16.806043 (+ 16us) server_negotiation.cc:407] Connection header received
0807 06:25:16.806084 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:16.806087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:16.806140 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:16.806232 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:16.807085 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.807704 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.809331 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.809696 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.809807 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:16.813405 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:16.813409 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:16.813412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:16.813438 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:16.815193 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.815749 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.815752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.815754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.815837 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.816180 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.816184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.816186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.816409 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:16.816638 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:16.816963 (+ 325us) server_negotiation.cc:299] Negotiation successful
0807 06:25:16.817093 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 06:25:16.819254 3145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 190c4aeb89cc48ce95ef424f465abd26: Bootstrap starting.
I20260807 06:25:16.821537 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:25:16.822770 3145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 190c4aeb89cc48ce95ef424f465abd26: Neither blocks nor log segments found. Creating new log.
I20260807 06:25:16.824098 3145 log.cc:824] T 00000000000000000000000000000000 P 190c4aeb89cc48ce95ef424f465abd26: Log is configured to *not* fsync() on all Append() calls
I20260807 06:25:16.835297 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.825331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43402 (local address 127.15.254.61:43345)
0807 06:25:16.825505 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:16.825510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:16.825920 (+ 410us) server_negotiation.cc:407] Connection header received
0807 06:25:16.826117 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:16.826121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:16.826173 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:16.826226 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:16.827150 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.827898 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.829114 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.829451 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.829526 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:16.831523 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:16.831528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:16.831531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:16.831563 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:16.833403 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.833907 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.833911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.833913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.834001 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.834285 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.834289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.834290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.834509 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:16.834623 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:16.834996 (+ 373us) server_negotiation.cc:299] Negotiation successful
0807 06:25:16.835121 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 06:25:16.836907 3151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.824813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35622 (local address 127.15.254.60:46753)
0807 06:25:16.824937 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:16.824943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:16.825890 (+ 947us) server_negotiation.cc:407] Connection header received
0807 06:25:16.826108 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:16.826115 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:16.826181 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:16.826247 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:16.827321 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.828143 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.829664 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.830115 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.830214 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:16.832449 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:16.832454 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:16.832459 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:16.832583 (+ 124us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:16.834579 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.835196 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.835201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.835204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.835303 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.835696 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.835699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.835701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.835959 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:16.836073 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:16.836571 (+ 498us) server_negotiation.cc:299] Negotiation successful
0807 06:25:16.836716 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260807 06:25:16.837950 2944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.824258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45140 (local address 127.15.254.62:37999)
0807 06:25:16.824457 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:16.824463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:16.825181 (+ 718us) server_negotiation.cc:407] Connection header received
0807 06:25:16.825331 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:16.825335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:16.825395 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:16.825485 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:16.826683 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.827601 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.828966 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.829365 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.829469 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:16.833388 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:16.833393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:16.833398 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:16.833439 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:16.835727 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.836289 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.836292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.836294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.836401 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.836785 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.836789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.836791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.837061 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:16.837195 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:16.837624 (+ 429us) server_negotiation.cc:299] Negotiation successful
0807 06:25:16.837765 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 06:25:16.840582 3145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 190c4aeb89cc48ce95ef424f465abd26: No bootstrap required, opened a new log
I20260807 06:25:16.844554 3145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 190c4aeb89cc48ce95ef424f465abd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62684b49ee54dfc83bd7cf7da4fc516" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37999 } } peers { permanent_uuid: "6103ac5e06ff48d185e311207c64c017" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 43345 } } peers { permanent_uuid: "190c4aeb89cc48ce95ef424f465abd26" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } }
I20260807 06:25:16.844805 3145 raft_consensus.cc:374] T 00000000000000000000000000000000 P 190c4aeb89cc48ce95ef424f465abd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:25:16.844893 3145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 190c4aeb89cc48ce95ef424f465abd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 190c4aeb89cc48ce95ef424f465abd26, State: Initialized, Role: FOLLOWER
I20260807 06:25:16.845448 3145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 190c4aeb89cc48ce95ef424f465abd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62684b49ee54dfc83bd7cf7da4fc516" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37999 } } peers { permanent_uuid: "6103ac5e06ff48d185e311207c64c017" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 43345 } } peers { permanent_uuid: "190c4aeb89cc48ce95ef424f465abd26" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } }
I20260807 06:25:16.846035 3189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190c4aeb89cc48ce95ef424f465abd26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62684b49ee54dfc83bd7cf7da4fc516" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37999 } } peers { permanent_uuid: "6103ac5e06ff48d185e311207c64c017" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 43345 } } peers { permanent_uuid: "190c4aeb89cc48ce95ef424f465abd26" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } } }
I20260807 06:25:16.846175 3189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190c4aeb89cc48ce95ef424f465abd26 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:25:16.847210 3145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 190c4aeb89cc48ce95ef424f465abd26 [sys.catalog]: configured and running, proceeding with master startup.
W20260807 06:25:16.853405 3205 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 190c4aeb89cc48ce95ef424f465abd26: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:25:16.853500 3205 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:25:16.851315 3054 master_runner.cc:186] Error getting master registration for 127.15.254.62:37999: OK, instance_id { permanent_uuid: "d62684b49ee54dfc83bd7cf7da4fc516" instance_seqno: 1786083916284537 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 06:25:16.854725 3054 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 06:25:16.922482 2939 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } attempt: 4
Aug 07 06:25:16 dist-test-slave-44d2 krb5kdc[2825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083916, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for kudu/127.15.254.60@KRBTEST.COM
I20260807 06:25:16.934005 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.923057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46753 (local address 127.0.0.1:35632)
0807 06:25:16.923283 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:16.923299 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:16.923427 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:16.923744 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:16.923748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:16.923991 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:16.924108 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:16.924115 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.925224 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.925228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:16.926087 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:16.926099 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.926709 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.926713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:16.926806 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:16.927479 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:16.927494 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:16.929935 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:16.932199 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:16.932203 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:16.932206 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:16.932514 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:16.932832 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:16.932836 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:16.932838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:16.932949 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:16.933423 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:16.933426 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:16.933538 (+ 112us) client_negotiation.cc:769] Sending connection context
0807 06:25:16.933653 (+ 115us) client_negotiation.cc:240] Negotiation successful
0807 06:25:16.933800 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260807 06:25:16.934051 3151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:16.923179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35632 (local address 127.15.254.60:46753)
0807 06:25:16.923309 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:16.923314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:16.923382 (+ 68us) server_negotiation.cc:407] Connection header received
0807 06:25:16.923541 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:16.923544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:16.923600 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:16.923702 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:16.924243 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.925097 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.926220 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:16.926591 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:16.926698 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:16.930101 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:16.930105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:16.930108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:16.930140 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:16.932068 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.932621 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.932625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.932627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.932728 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:16.933057 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:16.933060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:16.933062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:16.933311 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:16.933422 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:16.933735 (+ 313us) server_negotiation.cc:299] Negotiation successful
0807 06:25:16.933883 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 06:25:16.938213 2939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d62684b49ee54dfc83bd7cf7da4fc516: Bootstrap starting.
I20260807 06:25:16.941707 2939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d62684b49ee54dfc83bd7cf7da4fc516: Neither blocks nor log segments found. Creating new log.
I20260807 06:25:16.942907 2939 log.cc:824] T 00000000000000000000000000000000 P d62684b49ee54dfc83bd7cf7da4fc516: Log is configured to *not* fsync() on all Append() calls
I20260807 06:25:16.947196 2939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d62684b49ee54dfc83bd7cf7da4fc516: No bootstrap required, opened a new log
I20260807 06:25:16.951702 2939 raft_consensus.cc:348] T 00000000000000000000000000000000 P d62684b49ee54dfc83bd7cf7da4fc516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62684b49ee54dfc83bd7cf7da4fc516" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37999 } } peers { permanent_uuid: "6103ac5e06ff48d185e311207c64c017" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 43345 } } peers { permanent_uuid: "190c4aeb89cc48ce95ef424f465abd26" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } }
I20260807 06:25:16.952013 2939 raft_consensus.cc:374] T 00000000000000000000000000000000 P d62684b49ee54dfc83bd7cf7da4fc516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:25:16.952085 2939 raft_consensus.cc:729] T 00000000000000000000000000000000 P d62684b49ee54dfc83bd7cf7da4fc516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d62684b49ee54dfc83bd7cf7da4fc516, State: Initialized, Role: FOLLOWER
I20260807 06:25:16.952705 2939 consensus_queue.cc:260] T 00000000000000000000000000000000 P d62684b49ee54dfc83bd7cf7da4fc516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62684b49ee54dfc83bd7cf7da4fc516" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37999 } } peers { permanent_uuid: "6103ac5e06ff48d185e311207c64c017" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 43345 } } peers { permanent_uuid: "190c4aeb89cc48ce95ef424f465abd26" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } }
I20260807 06:25:16.953322 3211 sys_catalog.cc:455] T 00000000000000000000000000000000 P d62684b49ee54dfc83bd7cf7da4fc516 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62684b49ee54dfc83bd7cf7da4fc516" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37999 } } peers { permanent_uuid: "6103ac5e06ff48d185e311207c64c017" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 43345 } } peers { permanent_uuid: "190c4aeb89cc48ce95ef424f465abd26" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } } }
I20260807 06:25:16.953532 3211 sys_catalog.cc:458] T 00000000000000000000000000000000 P d62684b49ee54dfc83bd7cf7da4fc516 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:25:16.954268 2939 sys_catalog.cc:564] T 00000000000000000000000000000000 P d62684b49ee54dfc83bd7cf7da4fc516 [sys.catalog]: configured and running, proceeding with master startup.
W20260807 06:25:16.960386 3226 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d62684b49ee54dfc83bd7cf7da4fc516: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:25:16.960595 3226 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:25:16.963902 2843 master_runner.cc:263] No leader master found from master d62684b49ee54dfc83bd7cf7da4fc516
I20260807 06:25:16.963979 2843 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260807 06:25:17.009374 2972 tablet.cc:2369] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 06:25:17.009902 3080 tablet.cc:2369] T 00000000000000000000000000000000 P 190c4aeb89cc48ce95ef424f465abd26: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 06:25:17.030879 2870 tablet.cc:2369] T 00000000000000000000000000000000 P d62684b49ee54dfc83bd7cf7da4fc516: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:25:17.227655 3154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:25:17.227988 3154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62684b49ee54dfc83bd7cf7da4fc516" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37999 } } peers { permanent_uuid: "6103ac5e06ff48d185e311207c64c017" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 43345 } } peers { permanent_uuid: "190c4aeb89cc48ce95ef424f465abd26" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } }
I20260807 06:25:17.229246 3154 leader_election.cc:288] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d62684b49ee54dfc83bd7cf7da4fc516 (127.15.254.62:37999), 190c4aeb89cc48ce95ef424f465abd26 (127.15.254.60:46753)
I20260807 06:25:17.229967 2899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6103ac5e06ff48d185e311207c64c017" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d62684b49ee54dfc83bd7cf7da4fc516" is_pre_election: true
I20260807 06:25:17.230003 3109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6103ac5e06ff48d185e311207c64c017" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "190c4aeb89cc48ce95ef424f465abd26" is_pre_election: true
I20260807 06:25:17.230350 2899 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d62684b49ee54dfc83bd7cf7da4fc516 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6103ac5e06ff48d185e311207c64c017 in term 0.
I20260807 06:25:17.230350 3109 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 190c4aeb89cc48ce95ef424f465abd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6103ac5e06ff48d185e311207c64c017 in term 0.
I20260807 06:25:17.230861 2967 leader_election.cc:302] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6103ac5e06ff48d185e311207c64c017, d62684b49ee54dfc83bd7cf7da4fc516; no voters:
I20260807 06:25:17.231221 3154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:25:17.231359 3154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:25:17.231462 3154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:25:17.233117 3154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62684b49ee54dfc83bd7cf7da4fc516" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37999 } } peers { permanent_uuid: "6103ac5e06ff48d185e311207c64c017" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 43345 } } peers { permanent_uuid: "190c4aeb89cc48ce95ef424f465abd26" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } }
I20260807 06:25:17.233546 3154 leader_election.cc:288] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [CANDIDATE]: Term 1 election: Requested vote from peers d62684b49ee54dfc83bd7cf7da4fc516 (127.15.254.62:37999), 190c4aeb89cc48ce95ef424f465abd26 (127.15.254.60:46753)
I20260807 06:25:17.234028 3109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6103ac5e06ff48d185e311207c64c017" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "190c4aeb89cc48ce95ef424f465abd26"
I20260807 06:25:17.234028 2899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6103ac5e06ff48d185e311207c64c017" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d62684b49ee54dfc83bd7cf7da4fc516"
I20260807 06:25:17.234179 2899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d62684b49ee54dfc83bd7cf7da4fc516 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:25:17.234179 3109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 190c4aeb89cc48ce95ef424f465abd26 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:25:17.235528 2899 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d62684b49ee54dfc83bd7cf7da4fc516 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6103ac5e06ff48d185e311207c64c017 in term 1.
I20260807 06:25:17.235528 3109 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 190c4aeb89cc48ce95ef424f465abd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6103ac5e06ff48d185e311207c64c017 in term 1.
I20260807 06:25:17.235862 2969 leader_election.cc:302] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 190c4aeb89cc48ce95ef424f465abd26, 6103ac5e06ff48d185e311207c64c017; no voters:
I20260807 06:25:17.236104 3154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:25:17.236361 3154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [term 1 LEADER]: Becoming Leader. State: Replica: 6103ac5e06ff48d185e311207c64c017, State: Running, Role: LEADER
I20260807 06:25:17.236718 3154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62684b49ee54dfc83bd7cf7da4fc516" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37999 } } peers { permanent_uuid: "6103ac5e06ff48d185e311207c64c017" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 43345 } } peers { permanent_uuid: "190c4aeb89cc48ce95ef424f465abd26" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } }
I20260807 06:25:17.239161 3229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6103ac5e06ff48d185e311207c64c017. Latest consensus state: current_term: 1 leader_uuid: "6103ac5e06ff48d185e311207c64c017" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62684b49ee54dfc83bd7cf7da4fc516" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37999 } } peers { permanent_uuid: "6103ac5e06ff48d185e311207c64c017" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 43345 } } peers { permanent_uuid: "190c4aeb89cc48ce95ef424f465abd26" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } } }
I20260807 06:25:17.239300 3229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [sys.catalog]: This master's current role is: LEADER
I20260807 06:25:17.239734 3231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:25:17.240689 3231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:25:17.245400 3109 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 190c4aeb89cc48ce95ef424f465abd26 [term 1 FOLLOWER]: Refusing update from remote peer 6103ac5e06ff48d185e311207c64c017: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:25:17.245393 2899 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d62684b49ee54dfc83bd7cf7da4fc516 [term 1 FOLLOWER]: Refusing update from remote peer 6103ac5e06ff48d185e311207c64c017: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:25:17.246096 3154 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [LEADER]: Connected to new peer: Peer: permanent_uuid: "190c4aeb89cc48ce95ef424f465abd26" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:25:17.246326 3229 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d62684b49ee54dfc83bd7cf7da4fc516" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:25:17.250432 3189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190c4aeb89cc48ce95ef424f465abd26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6103ac5e06ff48d185e311207c64c017. Latest consensus state: current_term: 1 leader_uuid: "6103ac5e06ff48d185e311207c64c017" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62684b49ee54dfc83bd7cf7da4fc516" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37999 } } peers { permanent_uuid: "6103ac5e06ff48d185e311207c64c017" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 43345 } } peers { permanent_uuid: "190c4aeb89cc48ce95ef424f465abd26" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } } }
I20260807 06:25:17.250564 3189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190c4aeb89cc48ce95ef424f465abd26 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:25:17.252133 3211 sys_catalog.cc:455] T 00000000000000000000000000000000 P d62684b49ee54dfc83bd7cf7da4fc516 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6103ac5e06ff48d185e311207c64c017. Latest consensus state: current_term: 1 leader_uuid: "6103ac5e06ff48d185e311207c64c017" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62684b49ee54dfc83bd7cf7da4fc516" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37999 } } peers { permanent_uuid: "6103ac5e06ff48d185e311207c64c017" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 43345 } } peers { permanent_uuid: "190c4aeb89cc48ce95ef424f465abd26" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } } }
I20260807 06:25:17.252269 3211 sys_catalog.cc:458] T 00000000000000000000000000000000 P d62684b49ee54dfc83bd7cf7da4fc516 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:25:17.252861 3229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6103ac5e06ff48d185e311207c64c017" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62684b49ee54dfc83bd7cf7da4fc516" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37999 } } peers { permanent_uuid: "6103ac5e06ff48d185e311207c64c017" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 43345 } } peers { permanent_uuid: "190c4aeb89cc48ce95ef424f465abd26" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } } }
I20260807 06:25:17.252981 3229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [sys.catalog]: This master's current role is: LEADER
I20260807 06:25:17.253573 3154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6103ac5e06ff48d185e311207c64c017" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62684b49ee54dfc83bd7cf7da4fc516" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37999 } } peers { permanent_uuid: "6103ac5e06ff48d185e311207c64c017" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 43345 } } peers { permanent_uuid: "190c4aeb89cc48ce95ef424f465abd26" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } } }
I20260807 06:25:17.253690 3154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017 [sys.catalog]: This master's current role is: LEADER
I20260807 06:25:17.253988 3189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190c4aeb89cc48ce95ef424f465abd26 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6103ac5e06ff48d185e311207c64c017" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62684b49ee54dfc83bd7cf7da4fc516" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37999 } } peers { permanent_uuid: "6103ac5e06ff48d185e311207c64c017" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 43345 } } peers { permanent_uuid: "190c4aeb89cc48ce95ef424f465abd26" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } } }
I20260807 06:25:17.254112 3189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190c4aeb89cc48ce95ef424f465abd26 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:25:17.254446 3211 sys_catalog.cc:455] T 00000000000000000000000000000000 P d62684b49ee54dfc83bd7cf7da4fc516 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6103ac5e06ff48d185e311207c64c017" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62684b49ee54dfc83bd7cf7da4fc516" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37999 } } peers { permanent_uuid: "6103ac5e06ff48d185e311207c64c017" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 43345 } } peers { permanent_uuid: "190c4aeb89cc48ce95ef424f465abd26" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46753 } } }
I20260807 06:25:17.254568 3211 sys_catalog.cc:458] T 00000000000000000000000000000000 P d62684b49ee54dfc83bd7cf7da4fc516 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:25:17.255100 3231 catalog_manager.cc:1347] Generated new cluster ID: b458a432b1d84293ba2aa5cbf7987b29
I20260807 06:25:17.255167 3231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:25:17.256037 3236 mvcc.cc:205] Tried to move back new op lower bound from 7315799725030096896 to 7315799725004087296. Current Snapshot: MvccSnapshot[applied={T|T < 7315799725030096896}]
I20260807 06:25:17.269490 3231 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:25:17.270839 3231 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:25:17.287151 3231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6103ac5e06ff48d185e311207c64c017: Generated new TSK 0
I20260807 06:25:17.288050 3231 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:25:17.855273 3205 catalog_manager.cc:1259] Loaded cluster ID: b458a432b1d84293ba2aa5cbf7987b29
I20260807 06:25:17.855398 3205 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 190c4aeb89cc48ce95ef424f465abd26: loading cluster ID for follower catalog manager: success
I20260807 06:25:17.857513 3205 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 190c4aeb89cc48ce95ef424f465abd26: acquiring CA information for follower catalog manager: success
I20260807 06:25:17.858904 3205 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 190c4aeb89cc48ce95ef424f465abd26: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 06:25:17.962088 3226 catalog_manager.cc:1259] Loaded cluster ID: b458a432b1d84293ba2aa5cbf7987b29
I20260807 06:25:17.962184 3226 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d62684b49ee54dfc83bd7cf7da4fc516: loading cluster ID for follower catalog manager: success
I20260807 06:25:17.964061 3226 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d62684b49ee54dfc83bd7cf7da4fc516: acquiring CA information for follower catalog manager: success
I20260807 06:25:17.965394 3226 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d62684b49ee54dfc83bd7cf7da4fc516: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 06:25:18.353771 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 2843
W20260807 06:25:18.367515 2967 connection.cc:535] client connection to 127.15.254.62:37999 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 06:25:18.367590 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 2947
I20260807 06:25:18.383170 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 3054
2026-08-07T06:25:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3657 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:25:18 dist-test-slave-44d2 krb5kdc[3257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:25:18 dist-test-slave-44d2 krb5kdc[3257](info): set up 2 sockets
Aug 07 06:25:18 dist-test-slave-44d2 krb5kdc[3257](info): commencing operation
krb5kdc: starting...
W20260807 06:25:19.613188 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.200s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:25:19 dist-test-slave-44d2 krb5kdc[3257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:25:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:25:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:25:19.756234 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:37695
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:38897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:37695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:25:19.875612 3276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:19.875905 3276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:19.875946 3276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:19.880172 3276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:25:19.880242 3276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:19.880263 3276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:19.880280 3276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:25:19.880295 3276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:25:19.885721 3276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:38897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:37695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:37695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:19.886777 3276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:19.887792 3276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:19.896727 3282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:19.896730 3287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:19.896733 3283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:19.896967 3276 server_base.cc:1034] running on GCE node
I20260807 06:25:19.897604 3276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:19.898720 3276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:19.899904 3276 hybrid_clock.cc:648] HybridClock initialized: now 1786083919899873 us; error 42 us; skew 500 ppm
Aug 07 06:25:19 dist-test-slave-44d2 krb5kdc[3257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083919, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:19.905983 3276 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:25:19.906939 3276 webserver.cc:460] Webserver started at http://127.15.254.62:42987/ using document root <none> and password file <none>
I20260807 06:25:19.907522 3276 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:19.907611 3276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:25:19.907845 3276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:25:19.910012 3276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "a2bac808b3534a738daf5c24bf85be7a"
format_stamp: "Formatted at 2026-08-07 06:25:19 on dist-test-slave-44d2"
server_key: "a4f425710caaf51663e4b36f40525aaa"
server_key_iv: "5c3fb5a2e38ab635dd299f9687588cf5"
server_key_version: "encryptionkey@0"
I20260807 06:25:19.910588 3276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "a2bac808b3534a738daf5c24bf85be7a"
format_stamp: "Formatted at 2026-08-07 06:25:19 on dist-test-slave-44d2"
server_key: "a4f425710caaf51663e4b36f40525aaa"
server_key_iv: "5c3fb5a2e38ab635dd299f9687588cf5"
server_key_version: "encryptionkey@0"
I20260807 06:25:19.914623 3276 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:25:19.917714 3298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:19.919157 3276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:25:19.919324 3276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "a2bac808b3534a738daf5c24bf85be7a"
format_stamp: "Formatted at 2026-08-07 06:25:19 on dist-test-slave-44d2"
server_key: "a4f425710caaf51663e4b36f40525aaa"
server_key_iv: "5c3fb5a2e38ab635dd299f9687588cf5"
server_key_version: "encryptionkey@0"
I20260807 06:25:19.919467 3276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:19.935693 3276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:19.939365 3276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:19.939616 3276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:19.947252 3276 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:37695
I20260807 06:25:19.947265 3372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:37695 every 8 connection(s)
I20260807 06:25:19.948246 3276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:25:19.951261 3373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:25:19.954684 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 3276
I20260807 06:25:19.954838 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:25:19.955175 16376 external_mini_cluster.cc:1428] Setting key 8ede0f5b2680df3c49ce99456a787080
I20260807 06:25:19.958196 3373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2bac808b3534a738daf5c24bf85be7a: Bootstrap starting.
I20260807 06:25:19.960780 3373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2bac808b3534a738daf5c24bf85be7a: Neither blocks nor log segments found. Creating new log.
I20260807 06:25:19.961611 3373 log.cc:824] T 00000000000000000000000000000000 P a2bac808b3534a738daf5c24bf85be7a: Log is configured to *not* fsync() on all Append() calls
I20260807 06:25:19.963713 3373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2bac808b3534a738daf5c24bf85be7a: No bootstrap required, opened a new log
Aug 07 06:25:19 dist-test-slave-44d2 krb5kdc[3257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:25:19.967859 3373 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2bac808b3534a738daf5c24bf85be7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bac808b3534a738daf5c24bf85be7a" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37695 } }
I20260807 06:25:19.968158 3373 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2bac808b3534a738daf5c24bf85be7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:25:19.968238 3373 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2bac808b3534a738daf5c24bf85be7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2bac808b3534a738daf5c24bf85be7a, State: Initialized, Role: FOLLOWER
I20260807 06:25:19.968817 3373 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2bac808b3534a738daf5c24bf85be7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bac808b3534a738daf5c24bf85be7a" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37695 } }
I20260807 06:25:19.968982 3373 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2bac808b3534a738daf5c24bf85be7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:25:19.969058 3373 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2bac808b3534a738daf5c24bf85be7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:25:19.969184 3373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2bac808b3534a738daf5c24bf85be7a [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:25:19.970268 3373 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2bac808b3534a738daf5c24bf85be7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bac808b3534a738daf5c24bf85be7a" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37695 } }
I20260807 06:25:19.970479 3376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:19.956792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41174 (local address 127.15.254.62:37695)
0807 06:25:19.957370 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:19.957386 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:19.957438 (+ 52us) server_negotiation.cc:407] Connection header received
0807 06:25:19.958186 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:19.958207 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:19.958634 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:19.959001 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:19.959879 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:19.961021 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:19.961925 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:19.962407 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:19.962578 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:19.965249 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:19.965254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:19.965269 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:19.965297 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:19.967233 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:19.967797 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:19.967808 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:19.967813 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:19.967956 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:19.968265 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:19.968271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:19.968275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:19.968818 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:19.969057 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:19.969364 (+ 307us) server_negotiation.cc:299] Negotiation successful
0807 06:25:19.969670 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":190,"threads_started":1}
I20260807 06:25:19.970700 3373 leader_election.cc:302] T 00000000000000000000000000000000 P a2bac808b3534a738daf5c24bf85be7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2bac808b3534a738daf5c24bf85be7a; no voters:
I20260807 06:25:19.971110 3373 leader_election.cc:288] T 00000000000000000000000000000000 P a2bac808b3534a738daf5c24bf85be7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:25:19.971201 3378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2bac808b3534a738daf5c24bf85be7a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:25:19.971402 3378 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2bac808b3534a738daf5c24bf85be7a [term 1 LEADER]: Becoming Leader. State: Replica: a2bac808b3534a738daf5c24bf85be7a, State: Running, Role: LEADER
I20260807 06:25:19.972589 3373 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2bac808b3534a738daf5c24bf85be7a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:25:19.972759 3378 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2bac808b3534a738daf5c24bf85be7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bac808b3534a738daf5c24bf85be7a" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37695 } }
I20260807 06:25:19.973760 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:25:19.974570 3383 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2bac808b3534a738daf5c24bf85be7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2bac808b3534a738daf5c24bf85be7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bac808b3534a738daf5c24bf85be7a" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37695 } } }
I20260807 06:25:19.974721 3383 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2bac808b3534a738daf5c24bf85be7a [sys.catalog]: This master's current role is: LEADER
I20260807 06:25:19.975159 3382 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2bac808b3534a738daf5c24bf85be7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2bac808b3534a738daf5c24bf85be7a. Latest consensus state: current_term: 1 leader_uuid: "a2bac808b3534a738daf5c24bf85be7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2bac808b3534a738daf5c24bf85be7a" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 37695 } } }
I20260807 06:25:19.976069 3382 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2bac808b3534a738daf5c24bf85be7a [sys.catalog]: This master's current role is: LEADER
W20260807 06:25:19.978070 3396 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2bac808b3534a738daf5c24bf85be7a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:25:19.978161 3396 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:25:19.978237 3397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:25:19.978973 3397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:25:19.985000 3397 catalog_manager.cc:1347] Generated new cluster ID: d323ed64387c4f4eb23fefde16d2539a
I20260807 06:25:19.985083 3397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:25:20.004539 3397 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:25:20.006156 3397 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:25:20.013486 3397 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2bac808b3534a738daf5c24bf85be7a: Generated new TSK 0
I20260807 06:25:20.014456 3397 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:25:20.030607 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 3276
2026-08-07T06:25:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1646 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:25:20 dist-test-slave-44d2 krb5kdc[3405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:25:20 dist-test-slave-44d2 krb5kdc[3405](info): set up 2 sockets
Aug 07 06:25:20 dist-test-slave-44d2 krb5kdc[3405](info): commencing operation
krb5kdc: starting...
W20260807 06:25:21.274755 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.214s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:25:21 dist-test-slave-44d2 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:25:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:25:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:25:21.416356 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:39873
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:39873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:25:21.535046 3424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:21.535356 3424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:21.535439 3424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:21.539381 3424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:25:21.539475 3424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:21.539515 3424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:21.539558 3424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:25:21.539595 3424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:25:21.544893 3424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:39873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:39873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:21.545970 3424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:21.546989 3424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:21.555917 3435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:21.555967 3436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:21.555938 3438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:21.556185 3424 server_base.cc:1034] running on GCE node
I20260807 06:25:21.556891 3424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:21.558027 3424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:21.559195 3424 hybrid_clock.cc:648] HybridClock initialized: now 1786083921559166 us; error 45 us; skew 500 ppm
Aug 07 06:25:21 dist-test-slave-44d2 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083921, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:21.562006 3424 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:25:21.562942 3424 webserver.cc:460] Webserver started at http://127.15.254.62:38079/ using document root <none> and password file <none>
I20260807 06:25:21.563539 3424 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:21.563630 3424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:25:21.563862 3424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:25:21.565944 3424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "73e0944c72324f6b963b7ec680e23a13"
format_stamp: "Formatted at 2026-08-07 06:25:21 on dist-test-slave-44d2"
server_key: "87a96850453b58e181808917726ae5ac"
server_key_iv: "2941790fcdc1a49e43895ae38197a1cc"
server_key_version: "encryptionkey@0"
I20260807 06:25:21.566506 3424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "73e0944c72324f6b963b7ec680e23a13"
format_stamp: "Formatted at 2026-08-07 06:25:21 on dist-test-slave-44d2"
server_key: "87a96850453b58e181808917726ae5ac"
server_key_iv: "2941790fcdc1a49e43895ae38197a1cc"
server_key_version: "encryptionkey@0"
I20260807 06:25:21.570457 3424 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:25:21.572906 3449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:21.574265 3424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 06:25:21.574452 3424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "73e0944c72324f6b963b7ec680e23a13"
format_stamp: "Formatted at 2026-08-07 06:25:21 on dist-test-slave-44d2"
server_key: "87a96850453b58e181808917726ae5ac"
server_key_iv: "2941790fcdc1a49e43895ae38197a1cc"
server_key_version: "encryptionkey@0"
I20260807 06:25:21.574576 3424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:21.584336 3424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:21.587313 3424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:21.587520 3424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:21.595237 3424 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:39873
I20260807 06:25:21.595583 3526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:39873 every 8 connection(s)
I20260807 06:25:21.596406 3424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:25:21.599391 3527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:25:21.604171 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 3424
I20260807 06:25:21.604339 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:25:21.604708 16376 external_mini_cluster.cc:1428] Setting key ad83427a6f1172cbabaaa33d5840cf86
I20260807 06:25:21.605214 3527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13: Bootstrap starting.
I20260807 06:25:21.608186 3527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13: Neither blocks nor log segments found. Creating new log.
I20260807 06:25:21.609225 3527 log.cc:824] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13: Log is configured to *not* fsync() on all Append() calls
I20260807 06:25:21.611510 3527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13: No bootstrap required, opened a new log
Aug 07 06:25:21 dist-test-slave-44d2 krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:25:21.616219 3527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e0944c72324f6b963b7ec680e23a13" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39873 } }
I20260807 06:25:21.616540 3527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:25:21.616616 3527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73e0944c72324f6b963b7ec680e23a13, State: Initialized, Role: FOLLOWER
I20260807 06:25:21.617066 3527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e0944c72324f6b963b7ec680e23a13" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39873 } }
I20260807 06:25:21.617240 3527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:25:21.617337 3527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:25:21.617465 3527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:25:21.618409 3527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e0944c72324f6b963b7ec680e23a13" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39873 } }
I20260807 06:25:21.618772 3527 leader_election.cc:302] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73e0944c72324f6b963b7ec680e23a13; no voters:
I20260807 06:25:21.619133 3527 leader_election.cc:288] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:25:21.619272 3536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:25:21.619539 3536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [term 1 LEADER]: Becoming Leader. State: Replica: 73e0944c72324f6b963b7ec680e23a13, State: Running, Role: LEADER
I20260807 06:25:21.619877 3536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e0944c72324f6b963b7ec680e23a13" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39873 } }
I20260807 06:25:21.620682 3527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:25:21.621717 3537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73e0944c72324f6b963b7ec680e23a13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e0944c72324f6b963b7ec680e23a13" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39873 } } }
I20260807 06:25:21.621869 3537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [sys.catalog]: This master's current role is: LEADER
I20260807 06:25:21.621881 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:21.606350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36722 (local address 127.15.254.62:39873)
0807 06:25:21.606890 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:21.606911 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:21.607030 (+ 119us) server_negotiation.cc:407] Connection header received
0807 06:25:21.607951 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:21.607977 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:21.608444 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:21.608858 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:21.609611 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:21.611048 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:21.612045 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:21.612807 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:21.612999 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:21.615928 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:21.615936 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:21.615952 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:21.615987 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:21.618041 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:21.618496 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:21.618502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:21.618508 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:21.618675 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:21.618997 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:21.619000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:21.619002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:21.619611 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:21.620425 (+ 814us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:21.620606 (+ 181us) server_negotiation.cc:299] Negotiation successful
0807 06:25:21.620893 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":153,"threads_started":1}
I20260807 06:25:21.622367 3538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73e0944c72324f6b963b7ec680e23a13. Latest consensus state: current_term: 1 leader_uuid: "73e0944c72324f6b963b7ec680e23a13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e0944c72324f6b963b7ec680e23a13" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39873 } } }
I20260807 06:25:21.622498 3538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [sys.catalog]: This master's current role is: LEADER
I20260807 06:25:21.622843 3548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:25:21.626116 3548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:25:21.631914 3548 catalog_manager.cc:1347] Generated new cluster ID: dcbfd7a7a2494ebd942e46a350c95e10
I20260807 06:25:21.631999 3548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:25:21.670755 3548 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:25:21.671811 3548 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:25:21.679867 3548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13: Generated new TSK 0
I20260807 06:25:21.680759 3548 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
I20260807 06:25:21.717038 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:0
--local_ip_for_outbound_sockets=127.15.254.1
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:39873
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:25:21.837527 3562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:21.837853 3562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:21.838042 3562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:21.842224 3562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:21.842324 3562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:21.842423 3562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:25:21.848618 3562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:21.849788 3562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:21.850847 3562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:21.860873 3570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:21.860831 3571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:21.861053 3562 server_base.cc:1034] running on GCE node
W20260807 06:25:21.860888 3573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:21.861598 3562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:21.862291 3562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:21.863528 3562 hybrid_clock.cc:648] HybridClock initialized: now 1786083921863501 us; error 35 us; skew 500 ppm
Aug 07 06:25:21 dist-test-slave-44d2 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083921, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:21.866624 3562 init.cc:377] Logged in from keytab as kudu/127.15.254.1@KRBTEST.COM (short username kudu)
I20260807 06:25:21.867552 3562 webserver.cc:460] Webserver started at http://127.15.254.1:44317/ using document root <none> and password file <none>
I20260807 06:25:21.868111 3562 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:21.868173 3562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:25:21.868341 3562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:25:21.870522 3562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/data/instance:
uuid: "f3b70048f2a342b29cf159bc85beb641"
format_stamp: "Formatted at 2026-08-07 06:25:21 on dist-test-slave-44d2"
server_key: "589a60861abd78d620699cb968f18291"
server_key_iv: "d32854a5e4ca43d38a4de58900ac701b"
server_key_version: "encryptionkey@0"
I20260807 06:25:21.871052 3562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance:
uuid: "f3b70048f2a342b29cf159bc85beb641"
format_stamp: "Formatted at 2026-08-07 06:25:21 on dist-test-slave-44d2"
server_key: "589a60861abd78d620699cb968f18291"
server_key_iv: "d32854a5e4ca43d38a4de58900ac701b"
server_key_version: "encryptionkey@0"
I20260807 06:25:21.875034 3562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260807 06:25:21.877692 3581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:21.879314 3562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 06:25:21.879468 3562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "f3b70048f2a342b29cf159bc85beb641"
format_stamp: "Formatted at 2026-08-07 06:25:21 on dist-test-slave-44d2"
server_key: "589a60861abd78d620699cb968f18291"
server_key_iv: "d32854a5e4ca43d38a4de58900ac701b"
server_key_version: "encryptionkey@0"
I20260807 06:25:21.879576 3562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:21.906337 3562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:21.909499 3562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:21.909730 3562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:21.910429 3562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:25:21.911307 3562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:25:21.911355 3562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:21.911422 3562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:25:21.911453 3562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:21.921211 3562 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:33615
I20260807 06:25:21.921231 3750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:33615 every 8 connection(s)
I20260807 06:25:21.922367 3562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:25:21.926332 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 3562
I20260807 06:25:21.926445 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance
I20260807 06:25:21.926728 16376 external_mini_cluster.cc:1428] Setting key 72b04aac309752fc0a43b69342dba8bb
Aug 07 06:25:21 dist-test-slave-44d2 krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083921, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:25:21.937812 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:21.924517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:48309 (local address 127.15.254.62:39873)
0807 06:25:21.924649 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:21.924653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:21.925595 (+ 942us) server_negotiation.cc:407] Connection header received
0807 06:25:21.927177 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:21.927180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:21.927241 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:21.927349 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:21.928597 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:21.929139 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:21.930443 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:21.930682 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:21.930769 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:21.933837 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:21.933840 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:21.933842 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:21.933867 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:21.935344 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:21.936047 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:21.936050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:21.936051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:21.936130 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:21.936571 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:21.936575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:21.936577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:21.936757 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:21.936834 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:21.937568 (+ 734us) server_negotiation.cc:299] Negotiation successful
0807 06:25:21.937682 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.15.254.2@KRBTEST.COM; defaulting to no policy
I20260807 06:25:21.939131 3761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:21.924917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.1:48309)
0807 06:25:21.925436 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:21.925484 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:21.926959 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:21.927536 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:21.927544 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:21.927925 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:21.928424 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:21.928437 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:21.929269 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:21.929278 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:21.930289 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:21.930305 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:21.930845 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:21.930851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:21.931111 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:21.931775 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:21.931801 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:21.933660 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:21.935482 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:21.935492 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:21.935512 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:21.935927 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:21.936221 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:21.936227 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:21.936230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:21.936431 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:21.936869 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:21.936879 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:21.937263 (+ 384us) client_negotiation.cc:769] Sending connection context
0807 06:25:21.937506 (+ 243us) client_negotiation.cc:240] Negotiation successful
0807 06:25:21.937898 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":125,"threads_started":1}
I20260807 06:25:21.940621 3751 heartbeater.cc:347] Connected to a master server at 127.15.254.62:39873
I20260807 06:25:21.941043 3751 heartbeater.cc:464] Registering TS with master...
I20260807 06:25:21.941685 3751 heartbeater.cc:511] Master 127.15.254.62:39873 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.2@KRBTEST.COM" created.
I20260807 06:25:21.943569 3474 ts_manager.cc:194] Registered new tserver with Master: f3b70048f2a342b29cf159bc85beb641 (127.15.254.1:33615)
I20260807 06:25:21.944868 3474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.1@KRBTEST.COM'} at 127.15.254.1:48309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
I20260807 06:25:21.961496 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:0
--local_ip_for_outbound_sockets=127.15.254.2
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:39873
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:25:22.079416 3764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:22.079705 3764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:22.079778 3764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:22.083926 3764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:22.084071 3764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:22.084163 3764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:25:22.089481 3764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:22.090550 3764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:22.091588 3764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:22.101601 3775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:22.101608 3771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:22.101584 3772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:22.101660 3764 server_base.cc:1034] running on GCE node
I20260807 06:25:22.102171 3764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:22.102873 3764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:22.104060 3764 hybrid_clock.cc:648] HybridClock initialized: now 1786083922104042 us; error 29 us; skew 500 ppm
Aug 07 06:25:22 dist-test-slave-44d2 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083922, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:22.106833 3764 init.cc:377] Logged in from keytab as kudu/127.15.254.2@KRBTEST.COM (short username kudu)
I20260807 06:25:22.107674 3764 webserver.cc:460] Webserver started at http://127.15.254.2:41737/ using document root <none> and password file <none>
I20260807 06:25:22.108229 3764 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:22.108317 3764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:25:22.108631 3764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:25:22.110812 3764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/data/instance:
uuid: "f3a061febb17469f83a100cc0481e7c6"
format_stamp: "Formatted at 2026-08-07 06:25:22 on dist-test-slave-44d2"
server_key: "2dce174120ee0c7d898273c2ff0680aa"
server_key_iv: "de4a892d6a9086710c34fdd96c18c448"
server_key_version: "encryptionkey@0"
I20260807 06:25:22.111387 3764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance:
uuid: "f3a061febb17469f83a100cc0481e7c6"
format_stamp: "Formatted at 2026-08-07 06:25:22 on dist-test-slave-44d2"
server_key: "2dce174120ee0c7d898273c2ff0680aa"
server_key_iv: "de4a892d6a9086710c34fdd96c18c448"
server_key_version: "encryptionkey@0"
I20260807 06:25:22.115097 3764 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260807 06:25:22.117592 3789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:22.118844 3764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 06:25:22.118978 3764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "f3a061febb17469f83a100cc0481e7c6"
format_stamp: "Formatted at 2026-08-07 06:25:22 on dist-test-slave-44d2"
server_key: "2dce174120ee0c7d898273c2ff0680aa"
server_key_iv: "de4a892d6a9086710c34fdd96c18c448"
server_key_version: "encryptionkey@0"
I20260807 06:25:22.119107 3764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:22.136803 3764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:22.139986 3764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:22.140147 3764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:22.140805 3764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:25:22.141628 3764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:25:22.141685 3764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:22.141726 3764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:25:22.141741 3764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:22.151139 3764 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:46447
I20260807 06:25:22.151166 3959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:46447 every 8 connection(s)
I20260807 06:25:22.152026 3764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:25:22.159924 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 3764
I20260807 06:25:22.160076 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance
I20260807 06:25:22.160341 16376 external_mini_cluster.cc:1428] Setting key 07e43d6b0ac42657a3a859e8d52caa80
Aug 07 06:25:22 dist-test-slave-44d2 krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083922, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:25:22.167340 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:22.154029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:36893 (local address 127.15.254.62:39873)
0807 06:25:22.154166 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:22.154170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:22.154970 (+ 800us) server_negotiation.cc:407] Connection header received
0807 06:25:22.156382 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:22.156386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:22.156511 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:22.156603 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:22.157744 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.158284 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.159429 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.159692 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.159822 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:22.162869 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:22.162874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:22.162876 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:22.162910 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:22.164909 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:22.165445 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:22.165450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:22.165452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:22.165564 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:22.165973 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:22.165979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:22.165983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:22.166260 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:22.166372 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:22.166929 (+ 557us) server_negotiation.cc:299] Negotiation successful
0807 06:25:22.167081 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 06:25:22.168087 3965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:22.154356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.2:36893)
0807 06:25:22.154813 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:22.154860 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:22.156173 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:22.156776 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:22.156783 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:22.157137 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:22.157608 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:22.157621 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.158412 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.158416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:22.159296 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:22.159305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.159884 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.159887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:22.160104 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:22.160829 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:22.160846 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:22.162678 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:22.165043 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:22.165056 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:22.165068 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:22.165341 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:22.165685 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:22.165687 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:22.165688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:22.165827 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:22.166417 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:22.166423 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:22.166659 (+ 236us) client_negotiation.cc:769] Sending connection context
0807 06:25:22.166842 (+ 183us) client_negotiation.cc:240] Negotiation successful
0807 06:25:22.167141 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":118,"threads_started":1}
I20260807 06:25:22.169312 3961 heartbeater.cc:347] Connected to a master server at 127.15.254.62:39873
I20260807 06:25:22.169607 3961 heartbeater.cc:464] Registering TS with master...
I20260807 06:25:22.170229 3961 heartbeater.cc:511] Master 127.15.254.62:39873 requested a full tablet report, sending...
I20260807 06:25:22.171459 3474 ts_manager.cc:194] Registered new tserver with Master: f3a061febb17469f83a100cc0481e7c6 (127.15.254.2:46447)
I20260807 06:25:22.172041 3474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.2@KRBTEST.COM'} at 127.15.254.2:36893
WARNING: no policy specified for kudu/127.15.254.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
I20260807 06:25:22.192739 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:0
--local_ip_for_outbound_sockets=127.15.254.3
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:39873
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:25:22.310964 3968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:22.311249 3968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:22.311319 3968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:22.315335 3968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:22.315423 3968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:22.315505 3968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:25:22.321105 3968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:22.322363 3968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:22.323508 3968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:22.333464 3975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:22.333453 3974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:22.333464 3978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:22.333770 3968 server_base.cc:1034] running on GCE node
I20260807 06:25:22.334375 3968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:22.335057 3968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:22.336264 3968 hybrid_clock.cc:648] HybridClock initialized: now 1786083922336221 us; error 53 us; skew 500 ppm
Aug 07 06:25:22 dist-test-slave-44d2 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083922, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:22.339363 3968 init.cc:377] Logged in from keytab as kudu/127.15.254.3@KRBTEST.COM (short username kudu)
I20260807 06:25:22.340448 3968 webserver.cc:460] Webserver started at http://127.15.254.3:41491/ using document root <none> and password file <none>
I20260807 06:25:22.341128 3968 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:22.341235 3968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:25:22.341511 3968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:25:22.343782 3968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/data/instance:
uuid: "6bb1d215f13647b3a4f2e6a2797d443d"
format_stamp: "Formatted at 2026-08-07 06:25:22 on dist-test-slave-44d2"
server_key: "0db3d878711352931ebcb5772e68476d"
server_key_iv: "0045d4d5e3c9e1e0332ae6cf35c4e143"
server_key_version: "encryptionkey@0"
I20260807 06:25:22.344431 3968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance:
uuid: "6bb1d215f13647b3a4f2e6a2797d443d"
format_stamp: "Formatted at 2026-08-07 06:25:22 on dist-test-slave-44d2"
server_key: "0db3d878711352931ebcb5772e68476d"
server_key_iv: "0045d4d5e3c9e1e0332ae6cf35c4e143"
server_key_version: "encryptionkey@0"
I20260807 06:25:22.348608 3968 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 06:25:22.351395 3985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:22.352761 3968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:25:22.352900 3968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "6bb1d215f13647b3a4f2e6a2797d443d"
format_stamp: "Formatted at 2026-08-07 06:25:22 on dist-test-slave-44d2"
server_key: "0db3d878711352931ebcb5772e68476d"
server_key_iv: "0045d4d5e3c9e1e0332ae6cf35c4e143"
server_key_version: "encryptionkey@0"
I20260807 06:25:22.353031 3968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:22.374078 3968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:22.377220 3968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:22.377393 3968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:22.378062 3968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:25:22.378949 3968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:25:22.379001 3968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:22.379069 3968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:25:22.379102 3968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:22.388638 3968 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:36001
I20260807 06:25:22.388684 4158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:36001 every 8 connection(s)
I20260807 06:25:22.389567 3968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:25:22.390723 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 3968
I20260807 06:25:22.390848 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance
I20260807 06:25:22.391150 16376 external_mini_cluster.cc:1428] Setting key 2799f2525b3978b934969f5d04426d47
Aug 07 06:25:22 dist-test-slave-44d2 krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083922, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:25:22.404146 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:22.391740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:59569 (local address 127.15.254.62:39873)
0807 06:25:22.391919 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:22.391924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:22.393022 (+ 1098us) server_negotiation.cc:407] Connection header received
0807 06:25:22.394557 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:22.394561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:22.394615 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:22.394690 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:22.395869 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.396427 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.397630 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.397865 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.397981 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:22.400386 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:22.400390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:22.400392 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:22.400420 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:22.401966 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:22.402490 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:22.402494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:22.402496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:22.402583 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:22.403053 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:22.403056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:22.403057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:22.403227 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:22.403335 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:22.403834 (+ 499us) server_negotiation.cc:299] Negotiation successful
0807 06:25:22.403969 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 06:25:22.405072 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:22.392243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.3:59569)
0807 06:25:22.392874 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:22.392921 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:22.394322 (+ 1401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:22.394863 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:22.394870 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:22.395244 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:22.395721 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:22.395734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.396601 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.396604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:22.397505 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:22.397515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.398024 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.398028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:22.398193 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:22.398784 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:22.398800 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:22.400225 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:22.402088 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:22.402094 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:22.402108 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:22.402380 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:22.402673 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:22.402677 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:22.402679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:22.402820 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:22.403336 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:22.403344 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:22.403602 (+ 258us) client_negotiation.cc:769] Sending connection context
0807 06:25:22.403790 (+ 188us) client_negotiation.cc:240] Negotiation successful
0807 06:25:22.404110 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":243,"threads_started":1}
I20260807 06:25:22.406188 4161 heartbeater.cc:347] Connected to a master server at 127.15.254.62:39873
I20260807 06:25:22.406406 4161 heartbeater.cc:464] Registering TS with master...
I20260807 06:25:22.406841 4161 heartbeater.cc:511] Master 127.15.254.62:39873 requested a full tablet report, sending...
I20260807 06:25:22.407801 3474 ts_manager.cc:194] Registered new tserver with Master: 6bb1d215f13647b3a4f2e6a2797d443d (127.15.254.3:36001)
I20260807 06:25:22.408397 3474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.3@KRBTEST.COM'} at 127.15.254.3:59569
I20260807 06:25:22.417322 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:25:22.427006 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:22.419369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36732 (local address 127.15.254.62:39873)
0807 06:25:22.419484 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:22.419488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:22.419668 (+ 180us) server_negotiation.cc:407] Connection header received
0807 06:25:22.419760 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:22.419762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:22.419800 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:22.419866 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:22.420682 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.421239 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.422191 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.422511 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.422594 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:22.424040 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:22.424043 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:22.424045 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:22.424067 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:22.425490 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:22.425860 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:22.425863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:22.425864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:22.425984 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:22.426224 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:22.426226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:22.426227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:22.426462 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:22.426570 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:22.426762 (+ 192us) server_negotiation.cc:299] Negotiation successful
0807 06:25:22.426853 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260807 06:25:22.430039 3474 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36732:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:25:22.433120 3474 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:25:22.448402 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:22.443087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34350 (local address 127.15.254.1:33615)
0807 06:25:22.443703 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:22.443710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:22.443731 (+ 21us) server_negotiation.cc:407] Connection header received
0807 06:25:22.443803 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:22.443807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:22.443971 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:22.444337 (+ 366us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:25:22.444586 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.445353 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.446969 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.447691 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.447814 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:22.447983 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:22.448157 (+ 174us) server_negotiation.cc:299] Negotiation successful
0807 06:25:22.448250 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":88,"threads_started":1}
I20260807 06:25:22.448403 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:22.442828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:33615 (local address 127.0.0.1:34350)
0807 06:25:22.443337 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:22.443353 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:22.443453 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:22.444131 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:22.444135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:22.444171 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:25:22.444442 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:22.444453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.445483 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.445488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:22.446828 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:22.446844 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.447824 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.447828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:22.447947 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:22.447985 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 06:25:22.448113 (+ 128us) client_negotiation.cc:240] Negotiation successful
0807 06:25:22.448242 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":102,"spinlock_wait_cycles":4352,"thread_start_us":191,"threads_started":1}
I20260807 06:25:22.448794 4176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:22.442037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:36001 (local address 127.0.0.1:34860)
0807 06:25:22.442854 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:22.442913 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:22.443062 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:22.444118 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:22.444125 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:22.444162 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:25:22.444373 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:22.444383 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.445901 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.445904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:22.447107 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:22.447116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.448296 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.448300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:22.448409 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:22.448437 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 06:25:22.448535 (+ 98us) client_negotiation.cc:240] Negotiation successful
0807 06:25:22.448634 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":104,"spinlock_wait_cycles":11776,"thread_start_us":90,"threads_started":1}
I20260807 06:25:22.449265 4181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:22.442106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:46447 (local address 127.0.0.1:55084)
0807 06:25:22.443144 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:22.443158 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:22.443329 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:22.444163 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:22.444165 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:22.444177 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:25:22.444382 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:22.444388 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.445944 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.445949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:22.447163 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:22.447175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.448694 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.448699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:22.448785 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:22.448824 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 06:25:22.449046 (+ 222us) client_negotiation.cc:240] Negotiation successful
0807 06:25:22.449119 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"mutex_wait_us":138,"spinlock_wait_cycles":3328,"thread_start_us":91,"threads_started":1}
I20260807 06:25:22.449555 4179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:22.442218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34860 (local address 127.15.254.3:36001)
0807 06:25:22.443581 (+ 1363us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:22.443595 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:22.443620 (+ 25us) server_negotiation.cc:407] Connection header received
0807 06:25:22.443707 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:22.443713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:22.443917 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:22.444099 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:25:22.444539 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.445754 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.447237 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.448130 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.449040 (+ 910us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:22.449201 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:22.449257 (+ 56us) server_negotiation.cc:299] Negotiation successful
0807 06:25:22.449389 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1235,"thread_start_us":189,"threads_started":1}
I20260807 06:25:22.449836 4180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:22.442566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55084 (local address 127.15.254.2:46447)
0807 06:25:22.442925 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:22.442942 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:22.443248 (+ 306us) server_negotiation.cc:407] Connection header received
0807 06:25:22.443530 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:22.443536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:22.443766 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:22.443908 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:25:22.444539 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.445749 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.447341 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.448164 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.448277 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:22.448429 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:22.449442 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0807 06:25:22.449557 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":113,"threads_started":1}
I20260807 06:25:22.451982 3661 tablet_service.cc:1467] Processing CreateTablet for tablet 2c33a34d4d2d49c5963df99319b12022 (DEFAULT_TABLE table=test-table [id=afcf8d72f1fa40eeb29f253f949bd431]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:25:22.452131 4059 tablet_service.cc:1467] Processing CreateTablet for tablet 2c33a34d4d2d49c5963df99319b12022 (DEFAULT_TABLE table=test-table [id=afcf8d72f1fa40eeb29f253f949bd431]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:25:22.453197 3661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c33a34d4d2d49c5963df99319b12022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:25:22.453226 4059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c33a34d4d2d49c5963df99319b12022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:25:22.452929 3860 tablet_service.cc:1467] Processing CreateTablet for tablet 2c33a34d4d2d49c5963df99319b12022 (DEFAULT_TABLE table=test-table [id=afcf8d72f1fa40eeb29f253f949bd431]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:25:22.453926 3860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c33a34d4d2d49c5963df99319b12022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:25:22.460104 4183 tablet_bootstrap.cc:492] T 2c33a34d4d2d49c5963df99319b12022 P 6bb1d215f13647b3a4f2e6a2797d443d: Bootstrap starting.
I20260807 06:25:22.461443 4185 tablet_bootstrap.cc:492] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6: Bootstrap starting.
I20260807 06:25:22.462572 4183 tablet_bootstrap.cc:654] T 2c33a34d4d2d49c5963df99319b12022 P 6bb1d215f13647b3a4f2e6a2797d443d: Neither blocks nor log segments found. Creating new log.
I20260807 06:25:22.463567 4185 tablet_bootstrap.cc:654] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6: Neither blocks nor log segments found. Creating new log.
I20260807 06:25:22.463815 4183 log.cc:824] T 2c33a34d4d2d49c5963df99319b12022 P 6bb1d215f13647b3a4f2e6a2797d443d: Log is configured to *not* fsync() on all Append() calls
I20260807 06:25:22.464658 4184 tablet_bootstrap.cc:492] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641: Bootstrap starting.
I20260807 06:25:22.464660 4185 log.cc:824] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6: Log is configured to *not* fsync() on all Append() calls
I20260807 06:25:22.466208 4183 tablet_bootstrap.cc:492] T 2c33a34d4d2d49c5963df99319b12022 P 6bb1d215f13647b3a4f2e6a2797d443d: No bootstrap required, opened a new log
I20260807 06:25:22.466424 4183 ts_tablet_manager.cc:1397] T 2c33a34d4d2d49c5963df99319b12022 P 6bb1d215f13647b3a4f2e6a2797d443d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 06:25:22.466626 4184 tablet_bootstrap.cc:654] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641: Neither blocks nor log segments found. Creating new log.
I20260807 06:25:22.467334 4184 log.cc:824] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641: Log is configured to *not* fsync() on all Append() calls
I20260807 06:25:22.469556 4184 tablet_bootstrap.cc:492] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641: No bootstrap required, opened a new log
I20260807 06:25:22.469556 4185 tablet_bootstrap.cc:492] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6: No bootstrap required, opened a new log
I20260807 06:25:22.469762 4185 ts_tablet_manager.cc:1397] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260807 06:25:22.469762 4184 ts_tablet_manager.cc:1397] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260807 06:25:22.470082 4183 raft_consensus.cc:348] T 2c33a34d4d2d49c5963df99319b12022 P 6bb1d215f13647b3a4f2e6a2797d443d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:22.470323 4183 raft_consensus.cc:374] T 2c33a34d4d2d49c5963df99319b12022 P 6bb1d215f13647b3a4f2e6a2797d443d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:25:22.470386 4183 raft_consensus.cc:729] T 2c33a34d4d2d49c5963df99319b12022 P 6bb1d215f13647b3a4f2e6a2797d443d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bb1d215f13647b3a4f2e6a2797d443d, State: Initialized, Role: FOLLOWER
I20260807 06:25:22.470841 4183 consensus_queue.cc:260] T 2c33a34d4d2d49c5963df99319b12022 P 6bb1d215f13647b3a4f2e6a2797d443d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:22.471922 4183 ts_tablet_manager.cc:1428] T 2c33a34d4d2d49c5963df99319b12022 P 6bb1d215f13647b3a4f2e6a2797d443d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260807 06:25:22.472289 4161 heartbeater.cc:503] Master 127.15.254.62:39873 was elected leader, sending a full tablet report...
I20260807 06:25:22.474146 4185 raft_consensus.cc:348] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:22.474453 4185 raft_consensus.cc:374] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:25:22.474529 4185 raft_consensus.cc:729] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3a061febb17469f83a100cc0481e7c6, State: Initialized, Role: FOLLOWER
I20260807 06:25:22.474478 4184 raft_consensus.cc:348] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:22.474731 4184 raft_consensus.cc:374] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:25:22.474804 4184 raft_consensus.cc:729] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b70048f2a342b29cf159bc85beb641, State: Initialized, Role: FOLLOWER
I20260807 06:25:22.475018 4185 consensus_queue.cc:260] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:22.475203 4184 consensus_queue.cc:260] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:22.475859 3961 heartbeater.cc:503] Master 127.15.254.62:39873 was elected leader, sending a full tablet report...
I20260807 06:25:22.476104 4185 ts_tablet_manager.cc:1428] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260807 06:25:22.476104 4184 ts_tablet_manager.cc:1428] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260807 06:25:22.476442 3751 heartbeater.cc:503] Master 127.15.254.62:39873 was elected leader, sending a full tablet report...
I20260807 06:25:22.485893 4192 raft_consensus.cc:482] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:25:22.486053 4192 raft_consensus.cc:504] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:22.487243 4192 leader_election.cc:288] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bb1d215f13647b3a4f2e6a2797d443d (127.15.254.3:36001), f3b70048f2a342b29cf159bc85beb641 (127.15.254.1:33615)
I20260807 06:25:22.492780 4179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:22.488393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:46165 (local address 127.15.254.3:36001)
0807 06:25:22.488567 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:22.488575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:22.488594 (+ 19us) server_negotiation.cc:407] Connection header received
0807 06:25:22.488658 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:22.488660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:22.488723 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:22.488826 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:25:22.489267 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.490069 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.491580 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.492032 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.492130 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:22.492173 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:22.492571 (+ 398us) server_negotiation.cc:299] Negotiation successful
0807 06:25:22.492650 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 06:25:22.492777 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:22.488032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:58857 (local address 127.15.254.1:33615)
0807 06:25:22.488256 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:22.488261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:22.488452 (+ 191us) server_negotiation.cc:407] Connection header received
0807 06:25:22.488654 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:22.488658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:22.488737 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:22.488821 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:25:22.489260 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.490069 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.491650 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.492128 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.492263 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:22.492352 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:22.492530 (+ 178us) server_negotiation.cc:299] Negotiation successful
0807 06:25:22.492634 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260807 06:25:22.492806 4194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:22.487858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:33615 (local address 127.15.254.2:58857)
0807 06:25:22.488303 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:22.488337 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:22.488454 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:22.488953 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:22.488956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:22.488976 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:25:22.489121 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:22.489128 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.490220 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.490223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:22.491470 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:22.491481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.492258 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.492262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:22.492384 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:22.492409 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 06:25:22.492469 (+ 60us) client_negotiation.cc:240] Negotiation successful
0807 06:25:22.492619 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":34,"thread_start_us":154,"threads_started":1}
I20260807 06:25:22.492810 3965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:22.487532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:36001 (local address 127.15.254.2:46165)
0807 06:25:22.487687 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:22.487706 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:22.487820 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:22.488965 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:22.488968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:22.488989 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:25:22.489149 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:22.489157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.490203 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.490207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:22.491424 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:22.491452 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.492161 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.492168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:22.492366 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:22.492403 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 06:25:22.492547 (+ 144us) client_negotiation.cc:240] Negotiation successful
0807 06:25:22.492657 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":119,"spinlock_wait_cycles":12544}
I20260807 06:25:22.493518 3686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c33a34d4d2d49c5963df99319b12022" candidate_uuid: "f3a061febb17469f83a100cc0481e7c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3b70048f2a342b29cf159bc85beb641" is_pre_election: true
I20260807 06:25:22.493862 3686 raft_consensus.cc:2454] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3a061febb17469f83a100cc0481e7c6 in term 0.
I20260807 06:25:22.493952 4097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c33a34d4d2d49c5963df99319b12022" candidate_uuid: "f3a061febb17469f83a100cc0481e7c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" is_pre_election: true
I20260807 06:25:22.494210 4097 raft_consensus.cc:2454] T 2c33a34d4d2d49c5963df99319b12022 P 6bb1d215f13647b3a4f2e6a2797d443d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3a061febb17469f83a100cc0481e7c6 in term 0.
I20260807 06:25:22.494482 3791 leader_election.cc:302] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f3a061febb17469f83a100cc0481e7c6, f3b70048f2a342b29cf159bc85beb641; no voters:
I20260807 06:25:22.494768 4192 raft_consensus.cc:2781] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:25:22.494884 4192 raft_consensus.cc:482] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:25:22.494962 4192 raft_consensus.cc:3037] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:25:22.496176 4192 raft_consensus.cc:504] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:22.496606 4192 leader_election.cc:288] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6 [CANDIDATE]: Term 1 election: Requested vote from peers 6bb1d215f13647b3a4f2e6a2797d443d (127.15.254.3:36001), f3b70048f2a342b29cf159bc85beb641 (127.15.254.1:33615)
I20260807 06:25:22.496999 4097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c33a34d4d2d49c5963df99319b12022" candidate_uuid: "f3a061febb17469f83a100cc0481e7c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bb1d215f13647b3a4f2e6a2797d443d"
I20260807 06:25:22.497077 3686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c33a34d4d2d49c5963df99319b12022" candidate_uuid: "f3a061febb17469f83a100cc0481e7c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3b70048f2a342b29cf159bc85beb641"
I20260807 06:25:22.497139 4097 raft_consensus.cc:3037] T 2c33a34d4d2d49c5963df99319b12022 P 6bb1d215f13647b3a4f2e6a2797d443d [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:25:22.497182 3686 raft_consensus.cc:3037] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:25:22.498544 3686 raft_consensus.cc:2454] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3a061febb17469f83a100cc0481e7c6 in term 1.
I20260807 06:25:22.498544 4097 raft_consensus.cc:2454] T 2c33a34d4d2d49c5963df99319b12022 P 6bb1d215f13647b3a4f2e6a2797d443d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3a061febb17469f83a100cc0481e7c6 in term 1.
I20260807 06:25:22.498833 3791 leader_election.cc:302] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bb1d215f13647b3a4f2e6a2797d443d, f3a061febb17469f83a100cc0481e7c6; no voters:
I20260807 06:25:22.499003 4192 raft_consensus.cc:2781] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:25:22.499258 4192 raft_consensus.cc:686] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6 [term 1 LEADER]: Becoming Leader. State: Replica: f3a061febb17469f83a100cc0481e7c6, State: Running, Role: LEADER
I20260807 06:25:22.499637 4192 consensus_queue.cc:237] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:22.502785 3473 catalog_manager.cc:5495] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3a061febb17469f83a100cc0481e7c6 (127.15.254.2). New cstate: current_term: 1 leader_uuid: "f3a061febb17469f83a100cc0481e7c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } health_report { overall_health: UNKNOWN } } }
I20260807 06:25:22.529256 4180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:22.525678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55096 (local address 127.15.254.2:46447)
0807 06:25:22.525819 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:22.525823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:22.525838 (+ 15us) server_negotiation.cc:407] Connection header received
0807 06:25:22.526000 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:22.526003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:22.526063 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:22.526155 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:25:22.526384 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.526901 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.527754 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:22.527962 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:22.528036 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:22.528297 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:25:22.528794 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:25:22.528891 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:22.529028 (+ 137us) server_negotiation.cc:299] Negotiation successful
0807 06:25:22.529080 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 06:25:22.535394 3686 raft_consensus.cc:1261] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641 [term 1 FOLLOWER]: Refusing update from remote peer f3a061febb17469f83a100cc0481e7c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:25:22.535393 4097 raft_consensus.cc:1261] T 2c33a34d4d2d49c5963df99319b12022 P 6bb1d215f13647b3a4f2e6a2797d443d [term 1 FOLLOWER]: Refusing update from remote peer f3a061febb17469f83a100cc0481e7c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:25:22.536126 4192 consensus_queue.cc:1035] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:25:22.536316 4196 consensus_queue.cc:1035] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:25:22.546075 4201 mvcc.cc:205] Tried to move back new op lower bound from 7315799746697773056 to 7315799746560958464. Current Snapshot: MvccSnapshot[applied={T|T < 7315799746697773056}]
I20260807 06:25:22.553654 3474 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:25:22.553810 3474 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:25:22.556308 3474 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13: Sending DeleteTablet for 3 replicas of tablet 2c33a34d4d2d49c5963df99319b12022
I20260807 06:25:22.557237 3661 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c33a34d4d2d49c5963df99319b12022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:25:22 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:34350
I20260807 06:25:22.557240 4059 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c33a34d4d2d49c5963df99319b12022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:25:22 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:34860
I20260807 06:25:22.557372 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 3562
I20260807 06:25:22.558007 3860 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c33a34d4d2d49c5963df99319b12022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:25:22 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:55084
I20260807 06:25:22.567962 4214 tablet_replica.cc:330] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6: stopping tablet replica
I20260807 06:25:22.568362 4214 raft_consensus.cc:2229] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:25:22.568838 4214 raft_consensus.cc:2258] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:25:22.571489 4214 ts_tablet_manager.cc:1905] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:25:22.574911 4214 ts_tablet_manager.cc:1918] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:25:22.575016 4214 log.cc:1196] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/2c33a34d4d2d49c5963df99319b12022
I20260807 06:25:22.575318 4214 ts_tablet_manager.cc:1939] T 2c33a34d4d2d49c5963df99319b12022 P f3a061febb17469f83a100cc0481e7c6: Deleting consensus metadata
I20260807 06:25:22.575675 4212 tablet_replica.cc:330] T 2c33a34d4d2d49c5963df99319b12022 P 6bb1d215f13647b3a4f2e6a2797d443d: stopping tablet replica
I20260807 06:25:22.576049 4212 raft_consensus.cc:2229] T 2c33a34d4d2d49c5963df99319b12022 P 6bb1d215f13647b3a4f2e6a2797d443d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:25:22.576366 4212 raft_consensus.cc:2258] T 2c33a34d4d2d49c5963df99319b12022 P 6bb1d215f13647b3a4f2e6a2797d443d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:25:22.578222 4212 ts_tablet_manager.cc:1905] T 2c33a34d4d2d49c5963df99319b12022 P 6bb1d215f13647b3a4f2e6a2797d443d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:25:22.581527 4212 ts_tablet_manager.cc:1918] T 2c33a34d4d2d49c5963df99319b12022 P 6bb1d215f13647b3a4f2e6a2797d443d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:25:22.581624 4212 log.cc:1196] T 2c33a34d4d2d49c5963df99319b12022 P 6bb1d215f13647b3a4f2e6a2797d443d: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/2c33a34d4d2d49c5963df99319b12022
I20260807 06:25:22.581918 4212 ts_tablet_manager.cc:1939] T 2c33a34d4d2d49c5963df99319b12022 P 6bb1d215f13647b3a4f2e6a2797d443d: Deleting consensus metadata
I20260807 06:25:22.592516 3455 catalog_manager.cc:4841] TS 6bb1d215f13647b3a4f2e6a2797d443d (127.15.254.3:36001): tablet 2c33a34d4d2d49c5963df99319b12022 (table test-table [id=afcf8d72f1fa40eeb29f253f949bd431]) successfully deleted
I20260807 06:25:22.593523 3450 catalog_manager.cc:4841] TS f3a061febb17469f83a100cc0481e7c6 (127.15.254.2:46447): tablet 2c33a34d4d2d49c5963df99319b12022 (table test-table [id=afcf8d72f1fa40eeb29f253f949bd431]) successfully deleted
W20260807 06:25:22.601676 3455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.254.1:33615 (error 108)
I20260807 06:25:22.602093 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 3764
I20260807 06:25:22.609609 4181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:22.606938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:33615 (local address 127.0.0.1:34360)
0807 06:25:22.609374 (+ 2436us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:22.609482 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.1:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2312}
W20260807 06:25:22.610368 3455 catalog_manager.cc:4584] TS f3b70048f2a342b29cf159bc85beb641 (127.15.254.1:33615): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c33a34d4d2d49c5963df99319b12022: Network error: Client connection negotiation failed: client connection to 127.15.254.1:33615: connect: Connection refused (error 111)
I20260807 06:25:22.626142 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 3968
I20260807 06:25:22.644305 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 3424
I20260807 06:25:22.656594 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:39873
--webserver_interface=127.15.254.62
--webserver_port=38079
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:39873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:25:22.780295 4217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:22.780643 4217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:22.780692 4217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:22.784895 4217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:25:22.784960 4217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:22.784979 4217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:22.784996 4217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:25:22.785012 4217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:25:22.790383 4217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:39873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:39873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=38079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:22.791479 4217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:22.792513 4217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:22.809790 4227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:22.809815 4228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:22.809823 4217 server_base.cc:1034] running on GCE node
W20260807 06:25:22.809768 4231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:22.811040 4217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:22.812340 4217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:22.813555 4217 hybrid_clock.cc:648] HybridClock initialized: now 1786083922813532 us; error 36 us; skew 500 ppm
Aug 07 06:25:22 dist-test-slave-44d2 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083922, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:22.816749 4217 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:25:22.817828 4217 webserver.cc:460] Webserver started at http://127.15.254.62:38079/ using document root <none> and password file <none>
I20260807 06:25:22.818456 4217 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:22.818576 4217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:25:22.822805 4217 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260807 06:25:22.825423 4242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:22.826911 4217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 06:25:22.827080 4217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "73e0944c72324f6b963b7ec680e23a13"
format_stamp: "Formatted at 2026-08-07 06:25:21 on dist-test-slave-44d2"
server_key: "87a96850453b58e181808917726ae5ac"
server_key_iv: "2941790fcdc1a49e43895ae38197a1cc"
server_key_version: "encryptionkey@0"
I20260807 06:25:22.827721 4217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:22.849643 4217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:22.853626 4217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:22.853880 4217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:22.862445 4217 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:39873
I20260807 06:25:22.862759 4326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:39873 every 8 connection(s)
I20260807 06:25:22.863734 4217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:25:22.864717 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 4217
I20260807 06:25:22.865156 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:33615
--local_ip_for_outbound_sockets=127.15.254.1
--tserver_master_addrs=127.15.254.62:39873
--webserver_port=44317
--webserver_interface=127.15.254.1
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:25:22.868641 4327 sys_catalog.cc:263] Verifying existing consensus state
I20260807 06:25:22.870532 4327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13: Bootstrap starting.
I20260807 06:25:22.888996 4327 log.cc:824] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13: Log is configured to *not* fsync() on all Append() calls
I20260807 06:25:22.899029 4327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 06:25:22.899497 4327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13: Bootstrap complete.
I20260807 06:25:22.904206 4327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e0944c72324f6b963b7ec680e23a13" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39873 } }
I20260807 06:25:22.904706 4327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73e0944c72324f6b963b7ec680e23a13, State: Initialized, Role: FOLLOWER
I20260807 06:25:22.905256 4327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e0944c72324f6b963b7ec680e23a13" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39873 } }
I20260807 06:25:22.905378 4327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:25:22.905486 4327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:25:22.905612 4327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [term 1 FOLLOWER]: Advancing to term 2
I20260807 06:25:22.907675 4327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e0944c72324f6b963b7ec680e23a13" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39873 } }
I20260807 06:25:22.908053 4327 leader_election.cc:302] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73e0944c72324f6b963b7ec680e23a13; no voters:
I20260807 06:25:22.908416 4327 leader_election.cc:288] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 06:25:22.908561 4332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 06:25:22.908806 4332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [term 2 LEADER]: Becoming Leader. State: Replica: 73e0944c72324f6b963b7ec680e23a13, State: Running, Role: LEADER
I20260807 06:25:22.909129 4332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e0944c72324f6b963b7ec680e23a13" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39873 } }
I20260807 06:25:22.909828 4327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:25:22.910508 4333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "73e0944c72324f6b963b7ec680e23a13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e0944c72324f6b963b7ec680e23a13" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39873 } } }
I20260807 06:25:22.910640 4333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [sys.catalog]: This master's current role is: LEADER
I20260807 06:25:22.911141 4335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73e0944c72324f6b963b7ec680e23a13. Latest consensus state: current_term: 2 leader_uuid: "73e0944c72324f6b963b7ec680e23a13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e0944c72324f6b963b7ec680e23a13" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39873 } } }
I20260807 06:25:22.911262 4335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [sys.catalog]: This master's current role is: LEADER
I20260807 06:25:22.912096 4340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:25:22.917235 4340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:25:22.917857 4340 catalog_manager.cc:1259] Loaded cluster ID: dcbfd7a7a2494ebd942e46a350c95e10
I20260807 06:25:22.917925 4340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:25:22.919363 4340 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260807 06:25:22.923794 4340 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260807 06:25:22.929673 4340 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:25:22.930562 4340 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13: Loaded TSK: 0
I20260807 06:25:22.931856 4340 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 06:25:22.992401 4330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:22.992743 4330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:22.992789 4330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:22.996965 4330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:22.997036 4330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:22.997121 4330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:25:23.002485 4330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:33615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=44317
--tserver_master_addrs=127.15.254.62:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:23.003593 4330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:23.004722 4330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:23.016613 4368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:23.016628 4367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:23.016996 4330 server_base.cc:1034] running on GCE node
W20260807 06:25:23.016613 4371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:23.017552 4330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:23.018285 4330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:23.019482 4330 hybrid_clock.cc:648] HybridClock initialized: now 1786083923019454 us; error 46 us; skew 500 ppm
Aug 07 06:25:23 dist-test-slave-44d2 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083923, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:23.022651 4330 init.cc:377] Logged in from keytab as kudu/127.15.254.1@KRBTEST.COM (short username kudu)
I20260807 06:25:23.023738 4330 webserver.cc:460] Webserver started at http://127.15.254.1:44317/ using document root <none> and password file <none>
I20260807 06:25:23.024331 4330 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:23.024448 4330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:25:23.028614 4330 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:25:23.030882 4379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:23.032305 4330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:25:23.032423 4330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "f3b70048f2a342b29cf159bc85beb641"
format_stamp: "Formatted at 2026-08-07 06:25:21 on dist-test-slave-44d2"
server_key: "589a60861abd78d620699cb968f18291"
server_key_iv: "d32854a5e4ca43d38a4de58900ac701b"
server_key_version: "encryptionkey@0"
I20260807 06:25:23.033077 4330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:23.068626 4330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:23.072270 4330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:23.072517 4330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:23.073210 4330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:25:23.074407 4389 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260807 06:25:23.078701 4330 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 06:25:23.078766 4330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260807 06:25:23.078840 4330 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 06:25:23.080691 4330 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 06:25:23.080740 4330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260807 06:25:23.081046 4389 tablet_bootstrap.cc:492] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641: Bootstrap starting.
I20260807 06:25:23.092113 4330 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:33615
I20260807 06:25:23.092180 4543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:33615 every 8 connection(s)
I20260807 06:25:23.093305 4330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:25:23.094182 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 4330
I20260807 06:25:23.094672 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:46447
--local_ip_for_outbound_sockets=127.15.254.2
--tserver_master_addrs=127.15.254.62:39873
--webserver_port=41737
--webserver_interface=127.15.254.2
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:25:23.103669 4389 log.cc:824] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641: Log is configured to *not* fsync() on all Append() calls
Aug 07 06:25:23 dist-test-slave-44d2 krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083923, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:25:23.113229 4389 tablet_bootstrap.cc:492] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 06:25:23.113693 4389 tablet_bootstrap.cc:492] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641: Bootstrap complete.
I20260807 06:25:23.114317 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:23.096805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:47987 (local address 127.15.254.62:39873)
0807 06:25:23.097208 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:23.097231 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:23.097824 (+ 593us) server_negotiation.cc:407] Connection header received
0807 06:25:23.099491 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:23.099503 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:23.099816 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:23.100173 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:23.101409 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.102387 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.103573 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.104128 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.104315 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:23.107862 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:23.107869 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:23.107885 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:23.107919 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:23.110313 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:23.111109 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:23.111116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:23.111122 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:23.111289 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:23.111730 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:23.111734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:23.111736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:23.112195 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:23.112328 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:23.113059 (+ 731us) server_negotiation.cc:299] Negotiation successful
0807 06:25:23.113389 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":133,"threads_started":1}
I20260807 06:25:23.114343 4550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:23.096890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.1:47987)
0807 06:25:23.097611 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:23.097680 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:23.099014 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:23.100228 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:23.100236 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:23.100715 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:23.101216 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:23.101231 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.102561 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.102564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:23.103441 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:23.103450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.104302 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.104308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:23.104604 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:23.105387 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:23.105416 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:23.107624 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:23.110511 (+ 2887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:23.110519 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:23.110535 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:23.110967 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:23.111406 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:23.111410 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:23.111412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:23.111620 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:23.112331 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:23.112338 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:23.112686 (+ 348us) client_negotiation.cc:769] Sending connection context
0807 06:25:23.112879 (+ 193us) client_negotiation.cc:240] Negotiation successful
0807 06:25:23.113209 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":155,"threads_started":1}
I20260807 06:25:23.115315 4389 ts_tablet_manager.cc:1397] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641: Time spent bootstrapping tablet: real 0.034s user 0.021s sys 0.009s
I20260807 06:25:23.117801 4544 heartbeater.cc:347] Connected to a master server at 127.15.254.62:39873
I20260807 06:25:23.118047 4544 heartbeater.cc:464] Registering TS with master...
I20260807 06:25:23.118645 4544 heartbeater.cc:511] Master 127.15.254.62:39873 requested a full tablet report, sending...
I20260807 06:25:23.118211 4389 raft_consensus.cc:348] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:23.118739 4389 raft_consensus.cc:729] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b70048f2a342b29cf159bc85beb641, State: Initialized, Role: FOLLOWER
I20260807 06:25:23.119273 4389 consensus_queue.cc:260] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:23.120322 4389 ts_tablet_manager.cc:1428] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260807 06:25:23.122176 4264 ts_manager.cc:194] Registered new tserver with Master: f3b70048f2a342b29cf159bc85beb641 (127.15.254.1:33615)
I20260807 06:25:23.126281 4264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.1@KRBTEST.COM'} at 127.15.254.1:47987
I20260807 06:25:23.128804 4544 heartbeater.cc:503] Master 127.15.254.62:39873 was elected leader, sending a full tablet report...
Aug 07 06:25:23 dist-test-slave-44d2 krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083922, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for kudu/127.15.254.1@KRBTEST.COM
I20260807 06:25:23.137794 4553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:23.125703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:33615 (local address 127.0.0.1:34376)
0807 06:25:23.126450 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:23.126532 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:23.126695 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:23.127191 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:23.127196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:23.127639 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:23.127768 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:23.127776 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.129181 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.129185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:23.130116 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:23.130128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.131032 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.131036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:23.131129 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:23.131777 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:23.131799 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:23.133536 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:23.135640 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:23.135649 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:23.135654 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:23.135878 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:23.136283 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:23.136286 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:23.136287 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:23.136401 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:23.137093 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:23.137100 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:23.137305 (+ 205us) client_negotiation.cc:769] Sending connection context
0807 06:25:23.137440 (+ 135us) client_negotiation.cc:240] Negotiation successful
0807 06:25:23.137636 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":141,"threads_started":1}
I20260807 06:25:23.137903 4554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:23.125758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34376 (local address 127.15.254.1:33615)
0807 06:25:23.126186 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:23.126197 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:23.126631 (+ 434us) server_negotiation.cc:407] Connection header received
0807 06:25:23.126860 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:23.126866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:23.127041 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:23.127188 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:23.127912 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.129050 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.130384 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.130901 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.131019 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:23.133682 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:23.133693 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:23.133706 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:23.133734 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:23.135517 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:23.136045 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:23.136051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:23.136053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:23.136151 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:23.136552 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:23.136555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:23.136557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:23.136886 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:23.137031 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:23.137563 (+ 532us) server_negotiation.cc:299] Negotiation successful
0807 06:25:23.137741 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":120,"threads_started":1}
I20260807 06:25:23.139199 4448 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c33a34d4d2d49c5963df99319b12022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:25:22 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:34376
I20260807 06:25:23.140133 4555 tablet_replica.cc:330] stopping tablet replica
I20260807 06:25:23.140530 4555 raft_consensus.cc:2229] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:25:23.140799 4555 raft_consensus.cc:2258] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:25:23.141963 4555 ts_tablet_manager.cc:1905] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:25:23.145738 4555 ts_tablet_manager.cc:1918] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:25:23.145867 4555 log.cc:1196] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/wal/wals/2c33a34d4d2d49c5963df99319b12022
I20260807 06:25:23.146150 4555 ts_tablet_manager.cc:1939] T 2c33a34d4d2d49c5963df99319b12022 P f3b70048f2a342b29cf159bc85beb641: Deleting consensus metadata
I20260807 06:25:23.147892 4246 catalog_manager.cc:4841] TS f3b70048f2a342b29cf159bc85beb641 (127.15.254.1:33615): tablet 2c33a34d4d2d49c5963df99319b12022 (table test-table [id=afcf8d72f1fa40eeb29f253f949bd431]) successfully deleted
W20260807 06:25:23.226605 4548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:23.226864 4548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:23.226907 4548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:23.230924 4548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:23.231003 4548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:23.231118 4548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:25:23.236769 4548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:46447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=41737
--tserver_master_addrs=127.15.254.62:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:23.238050 4548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:23.239070 4548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 06:25:23.250801 4548 server_base.cc:1034] running on GCE node
W20260807 06:25:23.250751 4562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:23.250734 4566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:23.250759 4564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:23.251443 4548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:23.252101 4548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:23.253269 4548 hybrid_clock.cc:648] HybridClock initialized: now 1786083923253248 us; error 30 us; skew 500 ppm
Aug 07 06:25:23 dist-test-slave-44d2 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083923, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:23.256109 4548 init.cc:377] Logged in from keytab as kudu/127.15.254.2@KRBTEST.COM (short username kudu)
I20260807 06:25:23.257023 4548 webserver.cc:460] Webserver started at http://127.15.254.2:41737/ using document root <none> and password file <none>
I20260807 06:25:23.257555 4548 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:23.257627 4548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:25:23.261686 4548 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:25:23.263901 4577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:23.265378 4548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:25:23.265480 4548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "f3a061febb17469f83a100cc0481e7c6"
format_stamp: "Formatted at 2026-08-07 06:25:22 on dist-test-slave-44d2"
server_key: "2dce174120ee0c7d898273c2ff0680aa"
server_key_iv: "de4a892d6a9086710c34fdd96c18c448"
server_key_version: "encryptionkey@0"
I20260807 06:25:23.266040 4548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:23.279532 4548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:23.282995 4548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:23.283159 4548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:23.283720 4548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:25:23.284608 4548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:25:23.284658 4548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:23.284718 4548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:25:23.284771 4548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:23.294468 4548 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:46447
I20260807 06:25:23.294545 4736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:46447 every 8 connection(s)
I20260807 06:25:23.295643 4548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:25:23.305439 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 4548
I20260807 06:25:23.305928 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:36001
--local_ip_for_outbound_sockets=127.15.254.3
--tserver_master_addrs=127.15.254.62:39873
--webserver_port=41491
--webserver_interface=127.15.254.3
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 06:25:23 dist-test-slave-44d2 krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083923, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:25:23.311503 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:23.298172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:51791 (local address 127.15.254.62:39873)
0807 06:25:23.298379 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:23.298384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:23.299304 (+ 920us) server_negotiation.cc:407] Connection header received
0807 06:25:23.300777 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:23.300780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:23.300828 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:23.300904 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:23.302328 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.302866 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.304080 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.304310 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.304389 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:23.307209 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:23.307212 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:23.307214 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:23.307239 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:23.308981 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:23.309628 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:23.309632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:23.309633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:23.309720 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:23.310169 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:23.310171 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:23.310172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:23.310415 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:23.310499 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:23.311246 (+ 747us) server_negotiation.cc:299] Negotiation successful
0807 06:25:23.311373 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 06:25:23.312924 4739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:23.298602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.2:51791)
0807 06:25:23.299130 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:23.299175 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:23.300553 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:23.301253 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:23.301260 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:23.301681 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:23.302183 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:23.302197 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.302995 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.302998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:23.303962 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:23.303972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.304461 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.304464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:23.304633 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:23.305193 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:23.305210 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:23.307013 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:23.309125 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:23.309132 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:23.309147 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:23.309517 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:23.309820 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:23.309827 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:23.309831 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:23.310067 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:23.310538 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:23.310549 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:23.310913 (+ 364us) client_negotiation.cc:769] Sending connection context
0807 06:25:23.311181 (+ 268us) client_negotiation.cc:240] Negotiation successful
0807 06:25:23.311616 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":125,"threads_started":1}
I20260807 06:25:23.314302 4737 heartbeater.cc:347] Connected to a master server at 127.15.254.62:39873
I20260807 06:25:23.314576 4737 heartbeater.cc:464] Registering TS with master...
I20260807 06:25:23.315099 4737 heartbeater.cc:511] Master 127.15.254.62:39873 requested a full tablet report, sending...
I20260807 06:25:23.317256 4264 ts_manager.cc:194] Registered new tserver with Master: f3a061febb17469f83a100cc0481e7c6 (127.15.254.2:46447)
I20260807 06:25:23.317884 4264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.2@KRBTEST.COM'} at 127.15.254.2:51791
W20260807 06:25:23.434003 4740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:23.434306 4740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:23.434396 4740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:23.438782 4740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:23.438880 4740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:23.438974 4740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:25:23.444770 4740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:36001
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=41491
--tserver_master_addrs=127.15.254.62:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:23.446069 4740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:23.447187 4740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:23.457641 4746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:23.457733 4745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:23.457813 4740 server_base.cc:1034] running on GCE node
W20260807 06:25:23.457787 4748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:23.458357 4740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:23.459079 4740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:23.460304 4740 hybrid_clock.cc:648] HybridClock initialized: now 1786083923460266 us; error 49 us; skew 500 ppm
Aug 07 06:25:23 dist-test-slave-44d2 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083923, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:23.463496 4740 init.cc:377] Logged in from keytab as kudu/127.15.254.3@KRBTEST.COM (short username kudu)
I20260807 06:25:23.464661 4740 webserver.cc:460] Webserver started at http://127.15.254.3:41491/ using document root <none> and password file <none>
I20260807 06:25:23.465324 4740 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:23.465452 4740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:25:23.470162 4740 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260807 06:25:23.472684 4765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:23.474143 4740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 06:25:23.474344 4740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "6bb1d215f13647b3a4f2e6a2797d443d"
format_stamp: "Formatted at 2026-08-07 06:25:22 on dist-test-slave-44d2"
server_key: "0db3d878711352931ebcb5772e68476d"
server_key_iv: "0045d4d5e3c9e1e0332ae6cf35c4e143"
server_key_version: "encryptionkey@0"
I20260807 06:25:23.475178 4740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:23.499531 4740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:23.503460 4740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:23.503731 4740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:23.504550 4740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:25:23.505607 4740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:25:23.505686 4740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:23.505769 4740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:25:23.505825 4740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:23.516736 4740 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:36001
I20260807 06:25:23.516793 4929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:36001 every 8 connection(s)
I20260807 06:25:23.518075 4740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:25:23.527146 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 4740
Aug 07 06:25:23 dist-test-slave-44d2 krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083923, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:25:23.534991 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:23.520541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:48265 (local address 127.15.254.62:39873)
0807 06:25:23.520689 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:23.520694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:23.521726 (+ 1032us) server_negotiation.cc:407] Connection header received
0807 06:25:23.523278 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:23.523281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:23.523344 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:23.523458 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:23.524771 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.525322 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.526645 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.526912 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.527052 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:23.530467 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:23.530471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:23.530473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:23.530499 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:23.532175 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:23.532926 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:23.532930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:23.532932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:23.533050 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:23.533430 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:23.533434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:23.533436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:23.533693 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:23.533805 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:23.534596 (+ 791us) server_negotiation.cc:299] Negotiation successful
0807 06:25:23.534752 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 06:25:23.536146 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:23.520992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.3:48265)
0807 06:25:23.521547 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:23.521595 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:23.523006 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:23.523656 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:23.523663 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:23.524068 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:23.524610 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:23.524625 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.525492 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.525494 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:23.526472 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:23.526483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.527127 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.527131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:23.527324 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:23.528138 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:23.528162 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:23.530255 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:23.532302 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:23.532309 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:23.532328 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:23.532792 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:23.533156 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:23.533158 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:23.533160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:23.533314 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:23.533805 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:23.533811 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:23.534258 (+ 447us) client_negotiation.cc:769] Sending connection context
0807 06:25:23.534502 (+ 244us) client_negotiation.cc:240] Negotiation successful
0807 06:25:23.534921 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":141,"threads_started":1}
I20260807 06:25:23.537520 4930 heartbeater.cc:347] Connected to a master server at 127.15.254.62:39873
I20260807 06:25:23.537727 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:23.528033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36742 (local address 127.15.254.62:39873)
0807 06:25:23.528350 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:23.528355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:23.528506 (+ 151us) server_negotiation.cc:407] Connection header received
0807 06:25:23.528679 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:23.528682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:23.528740 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:23.528854 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:23.529895 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.530576 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.531685 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.532034 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.532150 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:23.533934 (+ 1784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:23.533938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:23.533940 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:23.533966 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:23.535671 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:23.536161 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:23.536167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:23.536172 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:23.536265 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:23.536637 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:23.536642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:23.536645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:23.536862 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:23.537100 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:23.537448 (+ 348us) server_negotiation.cc:299] Negotiation successful
0807 06:25:23.537568 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":115,"threads_started":1}
I20260807 06:25:23.537848 4930 heartbeater.cc:464] Registering TS with master...
I20260807 06:25:23.538404 4930 heartbeater.cc:511] Master 127.15.254.62:39873 requested a full tablet report, sending...
I20260807 06:25:23.539584 4263 ts_manager.cc:194] Registered new tserver with Master: 6bb1d215f13647b3a4f2e6a2797d443d (127.15.254.3:36001)
I20260807 06:25:23.540401 4263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.3@KRBTEST.COM'} at 127.15.254.3:48265
I20260807 06:25:23.540953 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:25:23.546581 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:23.541636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36756 (local address 127.15.254.62:39873)
0807 06:25:23.541764 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:23.541768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:23.542937 (+ 1169us) server_negotiation.cc:407] Connection header received
0807 06:25:23.543053 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:23.543055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:23.543144 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:23.543233 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:25:23.543412 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.543958 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.545121 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.545351 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.545447 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:23.545743 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:25:23.546162 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:25:23.546261 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:23.546349 (+ 88us) server_negotiation.cc:299] Negotiation successful
0807 06:25:23.546420 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 06:25:23.547317 4263 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:36756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:25:23.550861 4263 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:25:23.559396 4448 tablet_service.cc:1467] Processing CreateTablet for tablet 5663584b14f14367a1034f137f5e8e7e (DEFAULT_TABLE table=test-table [id=c1704b2982864359a729d09a01785b9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:25:23.560163 4448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5663584b14f14367a1034f137f5e8e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:25:23.563323 4389 tablet_bootstrap.cc:492] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641: Bootstrap starting.
I20260807 06:25:23.564319 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:23.558361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:46447 (local address 127.0.0.1:55106)
0807 06:25:23.559044 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:23.559058 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:23.559160 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:23.559825 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:23.559843 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:23.559864 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:25:23.559953 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:23.559959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.561504 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.561507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:23.562483 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:23.562493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.563824 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.563828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:23.563942 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:23.564000 (+ 58us) client_negotiation.cc:769] Sending connection context
0807 06:25:23.564080 (+ 80us) client_negotiation.cc:240] Negotiation successful
0807 06:25:23.564160 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":117,"thread_start_us":109,"threads_started":1}
I20260807 06:25:23.564360 4389 tablet_bootstrap.cc:654] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641: Neither blocks nor log segments found. Creating new log.
I20260807 06:25:23.564539 4553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:23.558059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:36001 (local address 127.0.0.1:34876)
0807 06:25:23.558247 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:23.558262 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:23.558387 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:23.559397 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:23.559400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:23.559426 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:25:23.559548 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:23.559558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.561451 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.561455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:23.562706 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:23.562717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.564115 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.564119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:23.564211 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:23.564231 (+ 20us) client_negotiation.cc:769] Sending connection context
0807 06:25:23.564278 (+ 47us) client_negotiation.cc:240] Negotiation successful
0807 06:25:23.564348 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":6}
I20260807 06:25:23.564873 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:23.558630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55106 (local address 127.15.254.2:46447)
0807 06:25:23.559006 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:23.559020 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:23.559157 (+ 137us) server_negotiation.cc:407] Connection header received
0807 06:25:23.559274 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:23.559279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:23.559473 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:23.559637 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:25:23.560075 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.561298 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.562620 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.563488 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.563590 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:23.563752 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:23.564595 (+ 843us) server_negotiation.cc:299] Negotiation successful
0807 06:25:23.564717 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":126,"threads_started":1}
I20260807 06:25:23.565207 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:23.558248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34876 (local address 127.15.254.3:36001)
0807 06:25:23.558774 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:23.558795 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:23.558824 (+ 29us) server_negotiation.cc:407] Connection header received
0807 06:25:23.558967 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:23.558977 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:23.559234 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:23.559426 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:25:23.559721 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.561298 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.562843 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.563860 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.564710 (+ 850us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:23.564871 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:23.564933 (+ 62us) server_negotiation.cc:299] Negotiation successful
0807 06:25:23.565056 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":126,"threads_started":1}
I20260807 06:25:23.566505 4389 tablet_bootstrap.cc:492] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641: No bootstrap required, opened a new log
I20260807 06:25:23.566632 4389 ts_tablet_manager.cc:1397] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 06:25:23.567091 4389 raft_consensus.cc:348] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:23.567235 4389 raft_consensus.cc:374] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:25:23.567287 4389 raft_consensus.cc:729] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b70048f2a342b29cf159bc85beb641, State: Initialized, Role: FOLLOWER
I20260807 06:25:23.567418 4389 consensus_queue.cc:260] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:23.567692 4389 ts_tablet_manager.cc:1428] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260807 06:25:23.568965 4649 tablet_service.cc:1467] Processing CreateTablet for tablet 5663584b14f14367a1034f137f5e8e7e (DEFAULT_TABLE table=test-table [id=c1704b2982864359a729d09a01785b9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:25:23.570410 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5663584b14f14367a1034f137f5e8e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:25:23.570410 4836 tablet_service.cc:1467] Processing CreateTablet for tablet 5663584b14f14367a1034f137f5e8e7e (DEFAULT_TABLE table=test-table [id=c1704b2982864359a729d09a01785b9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:25:23.571632 4836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5663584b14f14367a1034f137f5e8e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:25:23.579972 4949 tablet_bootstrap.cc:492] T 5663584b14f14367a1034f137f5e8e7e P 6bb1d215f13647b3a4f2e6a2797d443d: Bootstrap starting.
I20260807 06:25:23.579972 4948 tablet_bootstrap.cc:492] T 5663584b14f14367a1034f137f5e8e7e P f3a061febb17469f83a100cc0481e7c6: Bootstrap starting.
I20260807 06:25:23.582679 4949 tablet_bootstrap.cc:654] T 5663584b14f14367a1034f137f5e8e7e P 6bb1d215f13647b3a4f2e6a2797d443d: Neither blocks nor log segments found. Creating new log.
I20260807 06:25:23.583312 4948 tablet_bootstrap.cc:654] T 5663584b14f14367a1034f137f5e8e7e P f3a061febb17469f83a100cc0481e7c6: Neither blocks nor log segments found. Creating new log.
I20260807 06:25:23.584162 4949 log.cc:824] T 5663584b14f14367a1034f137f5e8e7e P 6bb1d215f13647b3a4f2e6a2797d443d: Log is configured to *not* fsync() on all Append() calls
I20260807 06:25:23.584162 4948 log.cc:824] T 5663584b14f14367a1034f137f5e8e7e P f3a061febb17469f83a100cc0481e7c6: Log is configured to *not* fsync() on all Append() calls
I20260807 06:25:23.586395 4948 tablet_bootstrap.cc:492] T 5663584b14f14367a1034f137f5e8e7e P f3a061febb17469f83a100cc0481e7c6: No bootstrap required, opened a new log
I20260807 06:25:23.586436 4949 tablet_bootstrap.cc:492] T 5663584b14f14367a1034f137f5e8e7e P 6bb1d215f13647b3a4f2e6a2797d443d: No bootstrap required, opened a new log
I20260807 06:25:23.586627 4948 ts_tablet_manager.cc:1397] T 5663584b14f14367a1034f137f5e8e7e P f3a061febb17469f83a100cc0481e7c6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260807 06:25:23.586683 4949 ts_tablet_manager.cc:1397] T 5663584b14f14367a1034f137f5e8e7e P 6bb1d215f13647b3a4f2e6a2797d443d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260807 06:25:23.589917 4948 raft_consensus.cc:348] T 5663584b14f14367a1034f137f5e8e7e P f3a061febb17469f83a100cc0481e7c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:23.590171 4948 raft_consensus.cc:374] T 5663584b14f14367a1034f137f5e8e7e P f3a061febb17469f83a100cc0481e7c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:25:23.590235 4948 raft_consensus.cc:729] T 5663584b14f14367a1034f137f5e8e7e P f3a061febb17469f83a100cc0481e7c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3a061febb17469f83a100cc0481e7c6, State: Initialized, Role: FOLLOWER
I20260807 06:25:23.590310 4949 raft_consensus.cc:348] T 5663584b14f14367a1034f137f5e8e7e P 6bb1d215f13647b3a4f2e6a2797d443d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:23.590588 4949 raft_consensus.cc:374] T 5663584b14f14367a1034f137f5e8e7e P 6bb1d215f13647b3a4f2e6a2797d443d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:25:23.590648 4949 raft_consensus.cc:729] T 5663584b14f14367a1034f137f5e8e7e P 6bb1d215f13647b3a4f2e6a2797d443d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bb1d215f13647b3a4f2e6a2797d443d, State: Initialized, Role: FOLLOWER
I20260807 06:25:23.590773 4948 consensus_queue.cc:260] T 5663584b14f14367a1034f137f5e8e7e P f3a061febb17469f83a100cc0481e7c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:23.591082 4949 consensus_queue.cc:260] T 5663584b14f14367a1034f137f5e8e7e P 6bb1d215f13647b3a4f2e6a2797d443d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:23.591567 4737 heartbeater.cc:503] Master 127.15.254.62:39873 was elected leader, sending a full tablet report...
I20260807 06:25:23.591876 4948 ts_tablet_manager.cc:1428] T 5663584b14f14367a1034f137f5e8e7e P f3a061febb17469f83a100cc0481e7c6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260807 06:25:23.592083 4949 ts_tablet_manager.cc:1428] T 5663584b14f14367a1034f137f5e8e7e P 6bb1d215f13647b3a4f2e6a2797d443d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260807 06:25:23.592466 4930 heartbeater.cc:503] Master 127.15.254.62:39873 was elected leader, sending a full tablet report...
W20260807 06:25:23.594626 4545 tablet.cc:2369] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:25:23.615423 4552 raft_consensus.cc:482] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:25:23.615681 4552 raft_consensus.cc:504] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:23.617039 4552 leader_election.cc:288] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bb1d215f13647b3a4f2e6a2797d443d (127.15.254.3:36001), f3a061febb17469f83a100cc0481e7c6 (127.15.254.2:46447)
I20260807 06:25:23.623236 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:23.617507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:41623 (local address 127.15.254.3:36001)
0807 06:25:23.617651 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:23.617656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:23.617841 (+ 185us) server_negotiation.cc:407] Connection header received
0807 06:25:23.618031 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:23.618034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:23.618080 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:23.618170 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:25:23.618574 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.619396 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.621608 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.622270 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.622402 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:23.622459 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:23.623012 (+ 553us) server_negotiation.cc:299] Negotiation successful
0807 06:25:23.623098 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 06:25:23.623236 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:23.617457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:33731 (local address 127.15.254.2:46447)
0807 06:25:23.617771 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:23.617778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:23.617917 (+ 139us) server_negotiation.cc:407] Connection header received
0807 06:25:23.618024 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:23.618027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:23.618080 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:23.618170 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:25:23.618575 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.619360 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.621615 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.622268 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.622402 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:23.622462 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:23.623023 (+ 561us) server_negotiation.cc:299] Negotiation successful
0807 06:25:23.623098 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260807 06:25:23.623311 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:23.617333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:46447 (local address 127.15.254.1:33731)
0807 06:25:23.617818 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:23.617832 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:23.617920 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:23.618197 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:23.618200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:23.618243 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:25:23.618446 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:23.618453 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.619552 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.619557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:23.621433 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:23.621448 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.622450 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.622457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:23.622704 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:23.622907 (+ 203us) client_negotiation.cc:769] Sending connection context
0807 06:25:23.622996 (+ 89us) client_negotiation.cc:240] Negotiation successful
0807 06:25:23.623056 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":380,"spinlock_wait_cycles":8576,"thread_start_us":120,"threads_started":1}
I20260807 06:25:23.623355 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:23.617333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:36001 (local address 127.15.254.1:41623)
0807 06:25:23.617731 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:23.617746 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:23.617881 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:23.618197 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:23.618200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:23.618242 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:25:23.618462 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:23.618467 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.619552 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.619557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:23.621451 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:23.621460 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.622460 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.622464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:23.622759 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:23.622826 (+ 67us) client_negotiation.cc:769] Sending connection context
0807 06:25:23.622939 (+ 113us) client_negotiation.cc:240] Negotiation successful
0807 06:25:23.623047 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":339,"spinlock_wait_cycles":31488,"thread_start_us":116,"threads_started":1}
I20260807 06:25:23.624218 4865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5663584b14f14367a1034f137f5e8e7e" candidate_uuid: "f3b70048f2a342b29cf159bc85beb641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" is_pre_election: true
I20260807 06:25:23.624219 4676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5663584b14f14367a1034f137f5e8e7e" candidate_uuid: "f3b70048f2a342b29cf159bc85beb641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3a061febb17469f83a100cc0481e7c6" is_pre_election: true
I20260807 06:25:23.624634 4865 raft_consensus.cc:2454] T 5663584b14f14367a1034f137f5e8e7e P 6bb1d215f13647b3a4f2e6a2797d443d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3b70048f2a342b29cf159bc85beb641 in term 0.
I20260807 06:25:23.624634 4676 raft_consensus.cc:2454] T 5663584b14f14367a1034f137f5e8e7e P f3a061febb17469f83a100cc0481e7c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3b70048f2a342b29cf159bc85beb641 in term 0.
I20260807 06:25:23.625227 4381 leader_election.cc:302] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6bb1d215f13647b3a4f2e6a2797d443d, f3a061febb17469f83a100cc0481e7c6, f3b70048f2a342b29cf159bc85beb641; no voters:
I20260807 06:25:23.625572 4552 raft_consensus.cc:2781] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:25:23.625684 4552 raft_consensus.cc:482] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:25:23.625746 4552 raft_consensus.cc:3037] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:25:23.627323 4552 raft_consensus.cc:504] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:23.627714 4552 leader_election.cc:288] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [CANDIDATE]: Term 1 election: Requested vote from peers 6bb1d215f13647b3a4f2e6a2797d443d (127.15.254.3:36001), f3a061febb17469f83a100cc0481e7c6 (127.15.254.2:46447)
I20260807 06:25:23.628170 4865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5663584b14f14367a1034f137f5e8e7e" candidate_uuid: "f3b70048f2a342b29cf159bc85beb641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bb1d215f13647b3a4f2e6a2797d443d"
I20260807 06:25:23.628196 4676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5663584b14f14367a1034f137f5e8e7e" candidate_uuid: "f3b70048f2a342b29cf159bc85beb641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3a061febb17469f83a100cc0481e7c6"
I20260807 06:25:23.628316 4865 raft_consensus.cc:3037] T 5663584b14f14367a1034f137f5e8e7e P 6bb1d215f13647b3a4f2e6a2797d443d [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:25:23.628348 4676 raft_consensus.cc:3037] T 5663584b14f14367a1034f137f5e8e7e P f3a061febb17469f83a100cc0481e7c6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:25:23.629501 4676 raft_consensus.cc:2454] T 5663584b14f14367a1034f137f5e8e7e P f3a061febb17469f83a100cc0481e7c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3b70048f2a342b29cf159bc85beb641 in term 1.
I20260807 06:25:23.629525 4865 raft_consensus.cc:2454] T 5663584b14f14367a1034f137f5e8e7e P 6bb1d215f13647b3a4f2e6a2797d443d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3b70048f2a342b29cf159bc85beb641 in term 1.
I20260807 06:25:23.629796 4381 leader_election.cc:302] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f3a061febb17469f83a100cc0481e7c6, f3b70048f2a342b29cf159bc85beb641; no voters:
I20260807 06:25:23.630000 4552 raft_consensus.cc:2781] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:25:23.630257 4552 raft_consensus.cc:686] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 1 LEADER]: Becoming Leader. State: Replica: f3b70048f2a342b29cf159bc85beb641, State: Running, Role: LEADER
I20260807 06:25:23.630527 4552 consensus_queue.cc:237] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:23.634099 4264 catalog_manager.cc:5495] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3b70048f2a342b29cf159bc85beb641 (127.15.254.1). New cstate: current_term: 1 leader_uuid: "f3b70048f2a342b29cf159bc85beb641" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } health_report { overall_health: HEALTHY } } }
I20260807 06:25:23.652662 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:23.648854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34380 (local address 127.15.254.1:33615)
0807 06:25:23.649152 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:23.649157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:23.649173 (+ 16us) server_negotiation.cc:407] Connection header received
0807 06:25:23.649263 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:23.649265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:23.649318 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:23.649461 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:25:23.649709 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.650233 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.651103 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:23.651341 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:23.651460 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:23.651749 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:25:23.652202 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:25:23.652334 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:23.652407 (+ 73us) server_negotiation.cc:299] Negotiation successful
0807 06:25:23.652466 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":88,"threads_started":1}
I20260807 06:25:23.656986 4676 raft_consensus.cc:1261] T 5663584b14f14367a1034f137f5e8e7e P f3a061febb17469f83a100cc0481e7c6 [term 1 FOLLOWER]: Refusing update from remote peer f3b70048f2a342b29cf159bc85beb641: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:25:23.657034 4865 raft_consensus.cc:1261] T 5663584b14f14367a1034f137f5e8e7e P 6bb1d215f13647b3a4f2e6a2797d443d [term 1 FOLLOWER]: Refusing update from remote peer f3b70048f2a342b29cf159bc85beb641: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:25:23.657774 4552 consensus_queue.cc:1035] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:25:23.657994 4957 consensus_queue.cc:1035] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:25:23.668669 4967 mvcc.cc:205] Tried to move back new op lower bound from 7315799751291666432 to 7315799751193911296. Current Snapshot: MvccSnapshot[applied={T|T < 7315799751291666432}]
I20260807 06:25:23.676453 4262 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:36756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:25:23.676726 4262 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:36756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:25:23.679867 4262 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13: Sending DeleteTablet for 3 replicas of tablet 5663584b14f14367a1034f137f5e8e7e
I20260807 06:25:23.680756 4448 tablet_service.cc:1514] Processing DeleteTablet for tablet 5663584b14f14367a1034f137f5e8e7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:25:23 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:34376
I20260807 06:25:23.680891 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet 5663584b14f14367a1034f137f5e8e7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:25:23 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:55106
I20260807 06:25:23.680989 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 4330
I20260807 06:25:23.686281 4981 tablet_replica.cc:330] T 5663584b14f14367a1034f137f5e8e7e P f3a061febb17469f83a100cc0481e7c6: stopping tablet replica
I20260807 06:25:23.686714 4981 raft_consensus.cc:2229] T 5663584b14f14367a1034f137f5e8e7e P f3a061febb17469f83a100cc0481e7c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:25:23.687074 4981 raft_consensus.cc:2258] T 5663584b14f14367a1034f137f5e8e7e P f3a061febb17469f83a100cc0481e7c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:25:23.689359 4981 ts_tablet_manager.cc:1905] T 5663584b14f14367a1034f137f5e8e7e P f3a061febb17469f83a100cc0481e7c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:25:23.690140 4836 tablet_service.cc:1514] Processing DeleteTablet for tablet 5663584b14f14367a1034f137f5e8e7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:25:23 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:34876
I20260807 06:25:23.691010 4982 tablet_replica.cc:330] T 5663584b14f14367a1034f137f5e8e7e P 6bb1d215f13647b3a4f2e6a2797d443d: stopping tablet replica
I20260807 06:25:23.691421 4982 raft_consensus.cc:2229] T 5663584b14f14367a1034f137f5e8e7e P 6bb1d215f13647b3a4f2e6a2797d443d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:25:23.691808 4982 raft_consensus.cc:2258] T 5663584b14f14367a1034f137f5e8e7e P 6bb1d215f13647b3a4f2e6a2797d443d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:25:23.692759 4981 ts_tablet_manager.cc:1918] T 5663584b14f14367a1034f137f5e8e7e P f3a061febb17469f83a100cc0481e7c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:25:23.692905 4981 log.cc:1196] T 5663584b14f14367a1034f137f5e8e7e P f3a061febb17469f83a100cc0481e7c6: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/5663584b14f14367a1034f137f5e8e7e
I20260807 06:25:23.693300 4981 ts_tablet_manager.cc:1939] T 5663584b14f14367a1034f137f5e8e7e P f3a061febb17469f83a100cc0481e7c6: Deleting consensus metadata
I20260807 06:25:23.693650 4982 ts_tablet_manager.cc:1905] T 5663584b14f14367a1034f137f5e8e7e P 6bb1d215f13647b3a4f2e6a2797d443d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:25:23.694633 4244 catalog_manager.cc:4841] TS f3a061febb17469f83a100cc0481e7c6 (127.15.254.2:46447): tablet 5663584b14f14367a1034f137f5e8e7e (table test-table [id=c1704b2982864359a729d09a01785b9f]) successfully deleted
I20260807 06:25:23.697638 4982 ts_tablet_manager.cc:1918] T 5663584b14f14367a1034f137f5e8e7e P 6bb1d215f13647b3a4f2e6a2797d443d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:25:23.697788 4982 log.cc:1196] T 5663584b14f14367a1034f137f5e8e7e P 6bb1d215f13647b3a4f2e6a2797d443d: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/5663584b14f14367a1034f137f5e8e7e
I20260807 06:25:23.698194 4982 ts_tablet_manager.cc:1939] T 5663584b14f14367a1034f137f5e8e7e P 6bb1d215f13647b3a4f2e6a2797d443d: Deleting consensus metadata
I20260807 06:25:23.699947 4246 catalog_manager.cc:4841] TS 6bb1d215f13647b3a4f2e6a2797d443d (127.15.254.3:36001): tablet 5663584b14f14367a1034f137f5e8e7e (table test-table [id=c1704b2982864359a729d09a01785b9f]) successfully deleted
W20260807 06:25:23.700649 4246 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.254.1:33615 (error 108)
I20260807 06:25:23.700955 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 4548
I20260807 06:25:23.707851 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:23.706066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:33615 (local address 127.0.0.1:34394)
0807 06:25:23.707630 (+ 1564us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:23.707732 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.1:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1433}
W20260807 06:25:23.709163 4246 catalog_manager.cc:4584] TS f3b70048f2a342b29cf159bc85beb641 (127.15.254.1:33615): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5663584b14f14367a1034f137f5e8e7e: Network error: Client connection negotiation failed: client connection to 127.15.254.1:33615: connect: Connection refused (error 111)
I20260807 06:25:23.719753 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 4740
I20260807 06:25:23.738067 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 4217
I20260807 06:25:23.754101 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:39873
--webserver_interface=127.15.254.62
--webserver_port=38079
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:39873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:25:23.872051 4984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:23.872334 4984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:23.872375 4984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:23.876421 4984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:25:23.876507 4984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:23.876538 4984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:23.876554 4984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:25:23.876569 4984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:25:23.882426 4984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:39873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:39873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=38079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:23.883618 4984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:23.884750 4984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:23.894333 4992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:23.894320 4990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:23.894711 4984 server_base.cc:1034] running on GCE node
W20260807 06:25:23.894320 4989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:23.895450 4984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:23.896781 4984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:23.898080 4984 hybrid_clock.cc:648] HybridClock initialized: now 1786083923898024 us; error 29 us; skew 500 ppm
Aug 07 06:25:23 dist-test-slave-44d2 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083923, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:23.901103 4984 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:25:23.902086 4984 webserver.cc:460] Webserver started at http://127.15.254.62:38079/ using document root <none> and password file <none>
I20260807 06:25:23.902621 4984 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:23.902680 4984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:25:23.906741 4984 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:25:23.909040 5005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:23.910459 4984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:25:23.910611 4984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "73e0944c72324f6b963b7ec680e23a13"
format_stamp: "Formatted at 2026-08-07 06:25:21 on dist-test-slave-44d2"
server_key: "87a96850453b58e181808917726ae5ac"
server_key_iv: "2941790fcdc1a49e43895ae38197a1cc"
server_key_version: "encryptionkey@0"
I20260807 06:25:23.911197 4984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:23.931384 4984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:23.934929 4984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:23.935240 4984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:23.943830 4984 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:39873
I20260807 06:25:23.944144 5084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:39873 every 8 connection(s)
I20260807 06:25:23.945178 4984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:25:23.950248 5085 sys_catalog.cc:263] Verifying existing consensus state
I20260807 06:25:23.952219 5085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13: Bootstrap starting.
I20260807 06:25:23.952472 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 4984
I20260807 06:25:23.953069 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:33615
--local_ip_for_outbound_sockets=127.15.254.1
--tserver_master_addrs=127.15.254.62:39873
--webserver_port=44317
--webserver_interface=127.15.254.1
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:25:23.973088 5085 log.cc:824] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13: Log is configured to *not* fsync() on all Append() calls
I20260807 06:25:23.985901 5085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 06:25:23.986382 5085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13: Bootstrap complete.
I20260807 06:25:23.991330 5085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e0944c72324f6b963b7ec680e23a13" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39873 } }
I20260807 06:25:23.991830 5085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73e0944c72324f6b963b7ec680e23a13, State: Initialized, Role: FOLLOWER
I20260807 06:25:23.992408 5085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e0944c72324f6b963b7ec680e23a13" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39873 } }
I20260807 06:25:23.992656 5085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:25:23.992764 5085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:25:23.992923 5085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [term 2 FOLLOWER]: Advancing to term 3
I20260807 06:25:23.995112 5085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e0944c72324f6b963b7ec680e23a13" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39873 } }
I20260807 06:25:23.995514 5085 leader_election.cc:302] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73e0944c72324f6b963b7ec680e23a13; no voters:
I20260807 06:25:23.995985 5085 leader_election.cc:288] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260807 06:25:23.996094 5091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 06:25:23.996326 5091 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [term 3 LEADER]: Becoming Leader. State: Replica: 73e0944c72324f6b963b7ec680e23a13, State: Running, Role: LEADER
I20260807 06:25:23.996701 5091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e0944c72324f6b963b7ec680e23a13" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39873 } }
I20260807 06:25:23.997422 5085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:25:23.998306 5092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "73e0944c72324f6b963b7ec680e23a13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e0944c72324f6b963b7ec680e23a13" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39873 } } }
I20260807 06:25:23.998476 5092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [sys.catalog]: This master's current role is: LEADER
I20260807 06:25:23.998809 5093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73e0944c72324f6b963b7ec680e23a13. Latest consensus state: current_term: 3 leader_uuid: "73e0944c72324f6b963b7ec680e23a13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e0944c72324f6b963b7ec680e23a13" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 39873 } } }
I20260807 06:25:23.998914 5093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73e0944c72324f6b963b7ec680e23a13 [sys.catalog]: This master's current role is: LEADER
I20260807 06:25:23.999500 5100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:25:24.005638 5100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:25:24.006448 5100 catalog_manager.cc:1259] Loaded cluster ID: dcbfd7a7a2494ebd942e46a350c95e10
I20260807 06:25:24.006525 5100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260807 06:25:24.007833 5100 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc0b93ea9ad google::LogMessage::Fail() at ??:0
@ 0x7fc0b93ed88a google::LogMessage::SendToLog() at ??:0
@ 0x7fc0b93ea4de google::LogMessage::Flush() at ??:0
@ 0x7fc0b93eba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc0c3d29fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fc0c3d2ad2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc0c3d27579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fc0c3d6b640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc0bb63c3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc0bb63cc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fc0bb63e3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc0bb62d512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc0bcc716db start_thread at ??:0
@ 0x7fc0b7ee861f clone at ??:0
W20260807 06:25:24.108405 5088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:24.108834 5088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:24.108908 5088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:24.115052 5088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:24.115136 5088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:24.115233 5088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:25:24.123178 5088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:33615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=44317
--tserver_master_addrs=127.15.254.62:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:24.124645 5088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:24.126055 5088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:24.139730 5142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:24.139730 5141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:24.140071 5144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:24.140393 5088 server_base.cc:1034] running on GCE node
I20260807 06:25:24.140874 5088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:24.141624 5088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:24.142856 5088 hybrid_clock.cc:648] HybridClock initialized: now 1786083924142806 us; error 63 us; skew 500 ppm
Aug 07 06:25:24 dist-test-slave-44d2 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083924, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:24.146555 5088 init.cc:377] Logged in from keytab as kudu/127.15.254.1@KRBTEST.COM (short username kudu)
I20260807 06:25:24.147687 5088 webserver.cc:460] Webserver started at http://127.15.254.1:44317/ using document root <none> and password file <none>
I20260807 06:25:24.148330 5088 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:24.148435 5088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:25:24.153056 5088 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:25:24.155891 5151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:24.157541 5088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:25:24.157702 5088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "f3b70048f2a342b29cf159bc85beb641"
format_stamp: "Formatted at 2026-08-07 06:25:21 on dist-test-slave-44d2"
server_key: "589a60861abd78d620699cb968f18291"
server_key_iv: "d32854a5e4ca43d38a4de58900ac701b"
server_key_version: "encryptionkey@0"
I20260807 06:25:24.158313 5088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:24.169706 5088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:24.173322 5088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:24.173552 5088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:24.174202 5088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:25:24.175377 5165 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260807 06:25:24.181066 5088 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 06:25:24.181156 5088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260807 06:25:24.181268 5088 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 06:25:24.183043 5088 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 06:25:24.183116 5088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260807 06:25:24.183425 5165 tablet_bootstrap.cc:492] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641: Bootstrap starting.
I20260807 06:25:24.194568 5088 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:33615
I20260807 06:25:24.194867 5317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:33615 every 8 connection(s)
I20260807 06:25:24.195796 5088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:25:24.202399 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 5088
I20260807 06:25:24.202863 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:46447
--local_ip_for_outbound_sockets=127.15.254.2
--tserver_master_addrs=127.15.254.62:39873
--webserver_port=41737
--webserver_interface=127.15.254.2
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:25:24.207855 5165 log.cc:824] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641: Log is configured to *not* fsync() on all Append() calls
I20260807 06:25:24.217862 5165 tablet_bootstrap.cc:492] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 06:25:24.218357 5165 tablet_bootstrap.cc:492] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641: Bootstrap complete.
I20260807 06:25:24.220072 5165 ts_tablet_manager.cc:1397] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641: Time spent bootstrapping tablet: real 0.037s user 0.021s sys 0.012s
I20260807 06:25:24.223464 5165 raft_consensus.cc:348] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:24.224030 5165 raft_consensus.cc:729] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b70048f2a342b29cf159bc85beb641, State: Initialized, Role: FOLLOWER
I20260807 06:25:24.224598 5165 consensus_queue.cc:260] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:24.225759 5165 ts_tablet_manager.cc:1428] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260807 06:25:24.268314 5321 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:25:24.198265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.1:35269)
0807 06:25:24.198828 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:24.198885 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:24.199755 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:24.267256 (+ 67501us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":163,"threads_started":1}
W20260807 06:25:24.269003 5318 heartbeater.cc:644] Failed to heartbeat to 127.15.254.62:39873 (0 consecutive failures): Network error: Failed to ping master at 127.15.254.62:39873: Client connection negotiation failed: client connection to 127.15.254.62:39873: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 06:25:24.269788 5321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:24.269494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.1:38887)
0807 06:25:24.269644 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:24.269721 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 06:25:24.270565 5321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:24.270332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.1:40293)
0807 06:25:24.270452 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:24.270505 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260807 06:25:24.270736 5318 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 06:25:24.330859 5322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:24.331141 5322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:24.331187 5322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:24.335389 5322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:24.335450 5322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:24.335556 5322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:25:24.340904 5322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:46447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=41737
--tserver_master_addrs=127.15.254.62:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:24.341954 5322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:24.343068 5322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:24.353567 5331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:24.353638 5334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:24.353607 5332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:24.353757 5322 server_base.cc:1034] running on GCE node
I20260807 06:25:24.354290 5322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:24.354979 5322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:24.356146 5322 hybrid_clock.cc:648] HybridClock initialized: now 1786083924356124 us; error 31 us; skew 500 ppm
Aug 07 06:25:24 dist-test-slave-44d2 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083924, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:24.359066 5322 init.cc:377] Logged in from keytab as kudu/127.15.254.2@KRBTEST.COM (short username kudu)
I20260807 06:25:24.360060 5322 webserver.cc:460] Webserver started at http://127.15.254.2:41737/ using document root <none> and password file <none>
I20260807 06:25:24.360773 5322 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:24.360873 5322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:25:24.365262 5322 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:25:24.367431 5343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:24.368767 5322 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 06:25:24.368911 5322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "f3a061febb17469f83a100cc0481e7c6"
format_stamp: "Formatted at 2026-08-07 06:25:22 on dist-test-slave-44d2"
server_key: "2dce174120ee0c7d898273c2ff0680aa"
server_key_iv: "de4a892d6a9086710c34fdd96c18c448"
server_key_version: "encryptionkey@0"
I20260807 06:25:24.369467 5322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:24.397403 5322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:24.400926 5322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:24.401155 5322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:24.401813 5322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:25:24.402709 5322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:25:24.402781 5322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:24.402854 5322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:25:24.402899 5322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:24.413054 5322 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:46447
I20260807 06:25:24.413097 5514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:46447 every 8 connection(s)
I20260807 06:25:24.414111 5322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:25:24.418567 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:24.416357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.2:41651)
0807 06:25:24.416990 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:24.417125 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":146,"threads_started":1}
W20260807 06:25:24.419093 5515 heartbeater.cc:644] Failed to heartbeat to 127.15.254.62:39873 (0 consecutive failures): Network error: Failed to ping master at 127.15.254.62:39873: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
I20260807 06:25:24.419808 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:24.419553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.2:47921)
0807 06:25:24.419677 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:24.419740 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260807 06:25:24.420584 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:24.420272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.2:50231)
0807 06:25:24.420408 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:24.420466 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260807 06:25:24.420770 5515 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 06:25:24.424361 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 5322
I20260807 06:25:24.424868 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:36001
--local_ip_for_outbound_sockets=127.15.254.3
--tserver_master_addrs=127.15.254.62:39873
--webserver_port=41491
--webserver_interface=127.15.254.3
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:25:24.543917 5520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:24.544207 5520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:24.544255 5520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:24.548398 5520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:24.548467 5520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:24.548632 5520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:25:24.554139 5520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:36001
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=41491
--tserver_master_addrs=127.15.254.62:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:24.555231 5520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:24.556268 5520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:24.566695 5529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:24.566695 5528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:24.566738 5532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:24.566963 5520 server_base.cc:1034] running on GCE node
I20260807 06:25:24.567480 5520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:24.568114 5520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:24.569310 5520 hybrid_clock.cc:648] HybridClock initialized: now 1786083924569289 us; error 48 us; skew 500 ppm
Aug 07 06:25:24 dist-test-slave-44d2 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083924, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:24.572208 5520 init.cc:377] Logged in from keytab as kudu/127.15.254.3@KRBTEST.COM (short username kudu)
I20260807 06:25:24.573220 5520 webserver.cc:460] Webserver started at http://127.15.254.3:41491/ using document root <none> and password file <none>
I20260807 06:25:24.573856 5520 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:24.573921 5520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:25:24.577878 5520 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:25:24.579975 5540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:24.581246 5520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 06:25:24.581389 5520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "6bb1d215f13647b3a4f2e6a2797d443d"
format_stamp: "Formatted at 2026-08-07 06:25:22 on dist-test-slave-44d2"
server_key: "0db3d878711352931ebcb5772e68476d"
server_key_iv: "0045d4d5e3c9e1e0332ae6cf35c4e143"
server_key_version: "encryptionkey@0"
I20260807 06:25:24.581964 5520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:24.594862 5520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:24.598297 5520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:24.598527 5520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:24.599170 5520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:25:24.600045 5520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:25:24.600093 5520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:24.600154 5520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:25:24.600188 5520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:24.609541 5520 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:36001
I20260807 06:25:24.609568 5724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:36001 every 8 connection(s)
I20260807 06:25:24.610579 5520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:25:24.613181 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 5520
I20260807 06:25:24.614920 5729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:24.612885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.3:52121)
0807 06:25:24.613498 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:24.614085 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":164,"threads_started":1}
W20260807 06:25:24.615348 5726 heartbeater.cc:644] Failed to heartbeat to 127.15.254.62:39873 (0 consecutive failures): Network error: Failed to ping master at 127.15.254.62:39873: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
I20260807 06:25:24.615890 5729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:24.615692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.3:44957)
0807 06:25:24.615795 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:24.615839 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260807 06:25:24.616438 5729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:24.616259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.3:44195)
0807 06:25:24.616353 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:24.616392 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260807 06:25:24.616671 5726 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 06:25:25.271967 5733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:25.271390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.1:58663)
0807 06:25:25.271681 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:25.271739 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":101,"threads_started":1}
I20260807 06:25:25.421880 5734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:25.421438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.2:48905)
0807 06:25:25.421701 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:25.421766 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":91,"threads_started":1}
I20260807 06:25:25.533766 5735 raft_consensus.cc:482] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:25:25.533994 5735 raft_consensus.cc:504] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:25.535440 5735 leader_election.cc:288] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bb1d215f13647b3a4f2e6a2797d443d (127.15.254.3:36001), f3a061febb17469f83a100cc0481e7c6 (127.15.254.2:46447)
Aug 07 06:25:25 dist-test-slave-44d2 krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083924, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for kudu/127.15.254.2@KRBTEST.COM
Aug 07 06:25:25 dist-test-slave-44d2 krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083924, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for kudu/127.15.254.3@KRBTEST.COM
I20260807 06:25:25.554484 5742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:25.535902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:57423 (local address 127.15.254.2:46447)
0807 06:25:25.536278 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:25.536301 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:25.536340 (+ 39us) server_negotiation.cc:407] Connection header received
0807 06:25:25.537116 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:25.537123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:25.537482 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:25.537797 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:25.539130 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:25.540132 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:25.541418 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:25.541896 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:25.542081 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:25.549691 (+ 7610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:25.549697 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:25.549713 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:25.549745 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:25.551730 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:25.552310 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:25.552316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:25.552321 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:25.552441 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:25.552886 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:25.552890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:25.552892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:25.553316 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:25.553465 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:25.553998 (+ 533us) server_negotiation.cc:299] Negotiation successful
0807 06:25:25.554270 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":104,"threads_started":1}
I20260807 06:25:25.554494 5733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:25.535645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:46447 (local address 127.15.254.1:57423)
0807 06:25:25.535844 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:25.535860 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:25.535973 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:25.537850 (+ 1877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:25.537856 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:25.538439 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:25.538958 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:25.538967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:25.540283 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:25.540292 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:25.541283 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:25.541294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:25.542033 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:25.542037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:25.542201 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:25.547423 (+ 5222us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:25.547451 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:25.549337 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:25.551870 (+ 2533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:25.551876 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:25.551890 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:25.552177 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:25.552626 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:25.552630 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:25.552632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:25.552783 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:25.553439 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:25.553446 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:25.553687 (+ 241us) client_negotiation.cc:769] Sending connection context
0807 06:25:25.553882 (+ 195us) client_negotiation.cc:240] Negotiation successful
0807 06:25:25.554242 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":61,"spinlock_wait_cycles":12928}
I20260807 06:25:25.554865 5743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:25.535913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:37309 (local address 127.15.254.3:36001)
0807 06:25:25.536367 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:25.536379 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:25.536424 (+ 45us) server_negotiation.cc:407] Connection header received
0807 06:25:25.537284 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:25.537291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:25.537628 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:25.537940 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:25.539147 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:25.540238 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:25.541435 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:25.541930 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:25.542164 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:25.550016 (+ 7852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:25.550021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:25.550033 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:25.550064 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:25.552058 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:25.552645 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:25.552651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:25.552655 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:25.552780 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:25.553126 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:25.553130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:25.553131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:25.553541 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:25.553704 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:25.554094 (+ 390us) server_negotiation.cc:299] Negotiation successful
0807 06:25:25.554412 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":131,"threads_started":1}
I20260807 06:25:25.555320 5741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:25.535645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:36001 (local address 127.15.254.1:37309)
0807 06:25:25.536047 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:25.536062 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:25.536147 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:25.537875 (+ 1728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:25.537878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:25.538450 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:25.538939 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:25.538956 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:25.540391 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:25.540395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:25.541287 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:25.541298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:25.542086 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:25.542090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:25.542204 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:25.547421 (+ 5217us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:25.547447 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:25.549735 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:25.552207 (+ 2472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:25.552212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:25.552215 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:25.552441 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:25.552899 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:25.552905 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:25.552910 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:25.553008 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:25.553672 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:25.553675 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:25.553787 (+ 112us) client_negotiation.cc:769] Sending connection context
0807 06:25:25.553903 (+ 116us) client_negotiation.cc:240] Negotiation successful
0807 06:25:25.554346 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":269,"spinlock_wait_cycles":30208,"thread_start_us":93,"threads_started":1}
W20260807 06:25:25.557320 5152 leader_election.cc:341] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3a061febb17469f83a100cc0481e7c6 (127.15.254.2:46447): Not found: Tablet not found: 5663584b14f14367a1034f137f5e8e7e
W20260807 06:25:25.557479 5153 leader_election.cc:341] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6bb1d215f13647b3a4f2e6a2797d443d (127.15.254.3:36001): Not found: Tablet not found: 5663584b14f14367a1034f137f5e8e7e
I20260807 06:25:25.557549 5152 leader_election.cc:302] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f3b70048f2a342b29cf159bc85beb641; no voters: 6bb1d215f13647b3a4f2e6a2797d443d, f3a061febb17469f83a100cc0481e7c6
I20260807 06:25:25.557839 5735 raft_consensus.cc:2726] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 06:25:25.617743 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:25.617275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.3:38501)
0807 06:25:25.617569 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:25.617624 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":124,"threads_started":1}
I20260807 06:25:26.273519 5745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:26.273063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.1:43887)
0807 06:25:26.273333 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:26.273392 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":90,"threads_started":1}
I20260807 06:25:26.423087 5746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:26.422646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.2:49863)
0807 06:25:26.422892 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:26.422998 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
I20260807 06:25:26.619191 5747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:26.618718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.3:50625)
0807 06:25:26.619008 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:26.619072 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":74,"threads_started":1}
I20260807 06:25:27.257350 5749 raft_consensus.cc:482] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:25:27.257494 5749 raft_consensus.cc:504] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:27.257967 5749 leader_election.cc:288] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bb1d215f13647b3a4f2e6a2797d443d (127.15.254.3:36001), f3a061febb17469f83a100cc0481e7c6 (127.15.254.2:46447)
W20260807 06:25:27.258876 5152 leader_election.cc:341] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3a061febb17469f83a100cc0481e7c6 (127.15.254.2:46447): Not found: Tablet not found: 5663584b14f14367a1034f137f5e8e7e
W20260807 06:25:27.259011 5153 leader_election.cc:341] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6bb1d215f13647b3a4f2e6a2797d443d (127.15.254.3:36001): Not found: Tablet not found: 5663584b14f14367a1034f137f5e8e7e
I20260807 06:25:27.259143 5153 leader_election.cc:302] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f3b70048f2a342b29cf159bc85beb641; no voters: 6bb1d215f13647b3a4f2e6a2797d443d, f3a061febb17469f83a100cc0481e7c6
I20260807 06:25:27.259310 5749 raft_consensus.cc:2726] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 06:25:27.274716 5751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:27.274303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.1:60435)
0807 06:25:27.274561 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:27.274617 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
I20260807 06:25:27.424309 5753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:27.423803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.2:50631)
0807 06:25:27.424153 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:27.424214 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":148,"threads_started":1}
I20260807 06:25:27.620460 5754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:27.619979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.3:48527)
0807 06:25:27.620270 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:27.620355 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":94,"threads_started":1}
I20260807 06:25:28.276088 5755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:28.275591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.1:58245)
0807 06:25:28.275930 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:28.275985 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":102,"threads_started":1}
I20260807 06:25:28.425577 5756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:28.425153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.2:43727)
0807 06:25:28.425421 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:28.425479 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":95,"threads_started":1}
I20260807 06:25:28.622207 5757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:28.621575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.3:50993)
0807 06:25:28.621953 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:28.622051 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":114,"threads_started":1}
I20260807 06:25:29.106170 5758 raft_consensus.cc:482] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:25:29.106356 5758 raft_consensus.cc:504] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:29.106809 5758 leader_election.cc:288] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bb1d215f13647b3a4f2e6a2797d443d (127.15.254.3:36001), f3a061febb17469f83a100cc0481e7c6 (127.15.254.2:46447)
W20260807 06:25:29.107734 5152 leader_election.cc:341] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3a061febb17469f83a100cc0481e7c6 (127.15.254.2:46447): Not found: Tablet not found: 5663584b14f14367a1034f137f5e8e7e
W20260807 06:25:29.107842 5153 leader_election.cc:341] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6bb1d215f13647b3a4f2e6a2797d443d (127.15.254.3:36001): Not found: Tablet not found: 5663584b14f14367a1034f137f5e8e7e
I20260807 06:25:29.108013 5153 leader_election.cc:302] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f3b70048f2a342b29cf159bc85beb641; no voters: 6bb1d215f13647b3a4f2e6a2797d443d, f3a061febb17469f83a100cc0481e7c6
I20260807 06:25:29.108237 5758 raft_consensus.cc:2726] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 06:25:29.277700 5761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:29.277197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.1:53837)
0807 06:25:29.277522 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:29.277590 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":83,"threads_started":1}
I20260807 06:25:29.426831 5762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:29.426384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.2:48477)
0807 06:25:29.426668 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:29.426730 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":82,"threads_started":1}
I20260807 06:25:29.623632 5763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:29.623125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.3:39517)
0807 06:25:29.623452 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:29.623519 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":105,"threads_started":1}
I20260807 06:25:30.279091 5767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:30.278560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.1:48183)
0807 06:25:30.278913 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:30.278981 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":100,"threads_started":1}
I20260807 06:25:30.428207 5768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:30.427730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.2:56893)
0807 06:25:30.428044 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:30.428107 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":103,"threads_started":1}
I20260807 06:25:30.625100 5770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:30.624619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.3:43665)
0807 06:25:30.624905 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:30.624986 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":97,"threads_started":1}
I20260807 06:25:31.280586 5772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:31.280013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.1:35077)
0807 06:25:31.280322 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:31.280387 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":109,"threads_started":1}
I20260807 06:25:31.429567 5774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:31.429069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.2:35131)
0807 06:25:31.429407 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:31.429469 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":99,"threads_started":1}
I20260807 06:25:31.626417 5775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:31.625897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.3:37789)
0807 06:25:31.626236 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:31.626298 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":79,"threads_started":1}
I20260807 06:25:32.281934 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:32.281469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.1:53889)
0807 06:25:32.281759 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:32.281819 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":82,"threads_started":1}
I20260807 06:25:32.431115 5782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:32.430568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.2:47667)
0807 06:25:32.430879 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:32.431004 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":84,"threads_started":1}
I20260807 06:25:32.627691 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:32.627246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.3:52885)
0807 06:25:32.627524 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:32.627584 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":85,"threads_started":1}
I20260807 06:25:32.782411 5785 raft_consensus.cc:482] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:25:32.782553 5785 raft_consensus.cc:504] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:32.783082 5785 leader_election.cc:288] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bb1d215f13647b3a4f2e6a2797d443d (127.15.254.3:36001), f3a061febb17469f83a100cc0481e7c6 (127.15.254.2:46447)
W20260807 06:25:32.783991 5152 leader_election.cc:341] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3a061febb17469f83a100cc0481e7c6 (127.15.254.2:46447): Not found: Tablet not found: 5663584b14f14367a1034f137f5e8e7e
W20260807 06:25:32.784111 5153 leader_election.cc:341] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6bb1d215f13647b3a4f2e6a2797d443d (127.15.254.3:36001): Not found: Tablet not found: 5663584b14f14367a1034f137f5e8e7e
I20260807 06:25:32.784224 5153 leader_election.cc:302] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f3b70048f2a342b29cf159bc85beb641; no voters: 6bb1d215f13647b3a4f2e6a2797d443d, f3a061febb17469f83a100cc0481e7c6
I20260807 06:25:32.784366 5785 raft_consensus.cc:2726] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 06:25:33.283396 5786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:33.282898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.1:47391)
0807 06:25:33.283225 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:33.283282 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":112,"threads_started":1}
I20260807 06:25:33.432427 5787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:33.431948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.2:55327)
0807 06:25:33.432256 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:33.432322 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":87,"threads_started":1}
I20260807 06:25:33.629062 5788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:33.628639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.3:40271)
0807 06:25:33.628904 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:33.628964 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":82,"threads_started":1}
I20260807 06:25:34.285023 5789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:34.284373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.1:54541)
0807 06:25:34.284769 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:34.284845 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":177,"threads_started":1}
I20260807 06:25:34.433733 5791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:34.433290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.2:37765)
0807 06:25:34.433577 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:34.433635 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":86,"threads_started":1}
I20260807 06:25:34.630404 5794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:34.629870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.3:44263)
0807 06:25:34.630225 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:34.630290 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":146,"threads_started":1}
I20260807 06:25:35.286442 5796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:35.285945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.1:44573)
0807 06:25:35.286241 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:35.286319 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":86,"threads_started":1}
I20260807 06:25:35.435153 5798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:35.434705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.2:58335)
0807 06:25:35.434991 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:35.435051 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":79,"threads_started":1}
I20260807 06:25:35.631848 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:35.631329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.3:53335)
0807 06:25:35.631659 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:35.631731 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":137,"threads_started":1}
I20260807 06:25:36.287734 5802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:36.287283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.1:60343)
0807 06:25:36.287561 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:36.287625 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":82,"threads_started":1}
I20260807 06:25:36.436630 5803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:36.436089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.2:41785)
0807 06:25:36.436381 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:36.436449 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":80,"threads_started":1}
I20260807 06:25:36.633252 5804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:36.632806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.3:43593)
0807 06:25:36.633089 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:36.633150 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":101,"threads_started":1}
I20260807 06:25:37.289227 5805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:37.288735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.1:45849)
0807 06:25:37.289061 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:37.289122 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":82,"threads_started":1}
I20260807 06:25:37.438112 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:37.437601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.2:46587)
0807 06:25:37.437929 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:37.438004 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":100,"threads_started":1}
I20260807 06:25:37.634748 5807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:37.634220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.3:58445)
0807 06:25:37.634575 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:37.634641 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":90,"threads_started":1}
I20260807 06:25:38.028030 5808 raft_consensus.cc:482] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:25:38.028178 5808 raft_consensus.cc:504] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1d215f13647b3a4f2e6a2797d443d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36001 } } peers { permanent_uuid: "f3a061febb17469f83a100cc0481e7c6" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 46447 } } peers { permanent_uuid: "f3b70048f2a342b29cf159bc85beb641" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 33615 } }
I20260807 06:25:38.028673 5808 leader_election.cc:288] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bb1d215f13647b3a4f2e6a2797d443d (127.15.254.3:36001), f3a061febb17469f83a100cc0481e7c6 (127.15.254.2:46447)
W20260807 06:25:38.029542 5153 leader_election.cc:341] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6bb1d215f13647b3a4f2e6a2797d443d (127.15.254.3:36001): Not found: Tablet not found: 5663584b14f14367a1034f137f5e8e7e
W20260807 06:25:38.029649 5152 leader_election.cc:341] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3a061febb17469f83a100cc0481e7c6 (127.15.254.2:46447): Not found: Tablet not found: 5663584b14f14367a1034f137f5e8e7e
I20260807 06:25:38.029779 5152 leader_election.cc:302] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f3b70048f2a342b29cf159bc85beb641; no voters: 6bb1d215f13647b3a4f2e6a2797d443d, f3a061febb17469f83a100cc0481e7c6
I20260807 06:25:38.029954 5808 raft_consensus.cc:2726] T 5663584b14f14367a1034f137f5e8e7e P f3b70048f2a342b29cf159bc85beb641 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 06:25:38.290485 5812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:38.290017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.1:55597)
0807 06:25:38.290297 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:38.290380 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":79,"threads_started":1}
I20260807 06:25:38.439369 5813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:38.438951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.2:50151)
0807 06:25:38.439213 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:38.439270 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":88,"threads_started":1}
I20260807 06:25:38.636220 5814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:38.635689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.3:39295)
0807 06:25:38.636040 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:38.636107 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":87,"threads_started":1}
I20260807 06:25:39.292043 5815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:39.291520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.1:40873)
0807 06:25:39.291803 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:39.291919 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":82,"threads_started":1}
I20260807 06:25:39.441023 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:39.440463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.2:46749)
0807 06:25:39.440845 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:39.440913 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":97,"threads_started":1}
I20260807 06:25:39.618443 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 5088
I20260807 06:25:39.637847 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:39.637422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:39873 (local address 127.15.254.3:34471)
0807 06:25:39.637689 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:39.637745 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":79,"threads_started":1}
I20260807 06:25:39.641662 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 5322
I20260807 06:25:39.653563 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 5520
2026-08-07T06:25:39Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19628 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:25:39 dist-test-slave-44d2 krb5kdc[5821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:25:39 dist-test-slave-44d2 krb5kdc[5821](info): set up 2 sockets
Aug 07 06:25:39 dist-test-slave-44d2 krb5kdc[5821](info): commencing operation
krb5kdc: starting...
W20260807 06:25:40.922420 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.233s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:25:41 dist-test-slave-44d2 krb5kdc[5821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:25:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:25:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:25:41.065888 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:32813
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:32813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:25:41.181473 5840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:41.181742 5840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:41.181782 5840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:41.185813 5840 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 06:25:41.185871 5840 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 06:25:41.185887 5840 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 06:25:41.185900 5840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:25:41.185918 5840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:41.185936 5840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:41.185952 5840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:25:41.185967 5840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:25:41.191152 5840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:32813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:32813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:41.192181 5840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:41.193282 5840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:41.202633 5851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:41.202715 5840 server_base.cc:1034] running on GCE node
W20260807 06:25:41.202672 5849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:41.202629 5846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:41.203506 5840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:41.204757 5840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:41.205991 5840 hybrid_clock.cc:648] HybridClock initialized: now 1786083941205933 us; error 41 us; skew 500 ppm
Aug 07 06:25:41 dist-test-slave-44d2 krb5kdc[5821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083941, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:41.209265 5840 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:25:41.210216 5840 webserver.cc:460] Webserver started at http://127.15.254.62:37901/ using document root <none> and password file <none>
I20260807 06:25:41.210820 5840 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:41.210916 5840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:25:41.211167 5840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:25:41.213544 5840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "7e23c9fb51f6476090bb2ddc94145062"
format_stamp: "Formatted at 2026-08-07 06:25:41 on dist-test-slave-44d2"
server_key: "011dd30a1b109ee21c771c289a0b3e39"
server_key_iv: "cec72b9b3e93328f135a40bd0edc4b9f"
server_key_version: "encryptionkey@0"
I20260807 06:25:41.214133 5840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "7e23c9fb51f6476090bb2ddc94145062"
format_stamp: "Formatted at 2026-08-07 06:25:41 on dist-test-slave-44d2"
server_key: "011dd30a1b109ee21c771c289a0b3e39"
server_key_iv: "cec72b9b3e93328f135a40bd0edc4b9f"
server_key_version: "encryptionkey@0"
I20260807 06:25:41.218160 5840 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:25:41.220817 5862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:41.222198 5840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 06:25:41.222363 5840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "7e23c9fb51f6476090bb2ddc94145062"
format_stamp: "Formatted at 2026-08-07 06:25:41 on dist-test-slave-44d2"
server_key: "011dd30a1b109ee21c771c289a0b3e39"
server_key_iv: "cec72b9b3e93328f135a40bd0edc4b9f"
server_key_version: "encryptionkey@0"
I20260807 06:25:41.222487 5840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:41.242841 5840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:41.246059 5840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:41.246286 5840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:41.253651 5840 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:32813
I20260807 06:25:41.253983 5943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:32813 every 8 connection(s)
I20260807 06:25:41.254678 5840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:25:41.257761 5944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:25:41.263563 5944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062: Bootstrap starting.
I20260807 06:25:41.263794 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 5840
I20260807 06:25:41.263958 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:25:41.264278 16376 external_mini_cluster.cc:1428] Setting key 2b37f920313ab4c8365d3602b0211413
I20260807 06:25:41.266476 5944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062: Neither blocks nor log segments found. Creating new log.
I20260807 06:25:41.267261 5944 log.cc:824] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062: Log is configured to *not* fsync() on all Append() calls
I20260807 06:25:41.269558 5944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062: No bootstrap required, opened a new log
Aug 07 06:25:41 dist-test-slave-44d2 krb5kdc[5821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:25:41.274356 5944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e23c9fb51f6476090bb2ddc94145062" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 32813 } }
I20260807 06:25:41.274636 5944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:25:41.274718 5944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e23c9fb51f6476090bb2ddc94145062, State: Initialized, Role: FOLLOWER
I20260807 06:25:41.275205 5944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e23c9fb51f6476090bb2ddc94145062" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 32813 } }
I20260807 06:25:41.275362 5944 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:25:41.275454 5944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:25:41.275583 5944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:25:41.276647 5944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e23c9fb51f6476090bb2ddc94145062" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 32813 } }
I20260807 06:25:41.277019 5944 leader_election.cc:302] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e23c9fb51f6476090bb2ddc94145062; no voters:
I20260807 06:25:41.277408 5944 leader_election.cc:288] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:25:41.277568 5951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:25:41.277842 5951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [term 1 LEADER]: Becoming Leader. State: Replica: 7e23c9fb51f6476090bb2ddc94145062, State: Running, Role: LEADER
I20260807 06:25:41.278841 5944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:25:41.278270 5951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e23c9fb51f6476090bb2ddc94145062" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 32813 } }
I20260807 06:25:41.280821 5948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:41.265761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51368 (local address 127.15.254.62:32813)
0807 06:25:41.266256 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:41.266276 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:41.266313 (+ 37us) server_negotiation.cc:407] Connection header received
0807 06:25:41.267096 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:41.267117 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:41.267521 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:41.267905 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:41.268855 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:41.269946 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:41.271134 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:41.271644 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:41.271774 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:41.274906 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:41.274911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:41.274927 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:41.275013 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:41.277162 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:41.277742 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:41.277748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:41.277755 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:41.277901 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:41.278260 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:41.278263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:41.278265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:41.278717 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:41.278887 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:41.279367 (+ 480us) server_negotiation.cc:299] Negotiation successful
0807 06:25:41.279689 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":151,"threads_started":1}
I20260807 06:25:41.281073 5952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e23c9fb51f6476090bb2ddc94145062" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e23c9fb51f6476090bb2ddc94145062" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 32813 } } }
I20260807 06:25:41.281189 5952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [sys.catalog]: This master's current role is: LEADER
I20260807 06:25:41.282162 5955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e23c9fb51f6476090bb2ddc94145062. Latest consensus state: current_term: 1 leader_uuid: "7e23c9fb51f6476090bb2ddc94145062" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e23c9fb51f6476090bb2ddc94145062" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 32813 } } }
I20260807 06:25:41.282267 5955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [sys.catalog]: This master's current role is: LEADER
W20260807 06:25:41.284374 5971 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:25:41.284499 5971 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:25:41.284631 5976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:25:41.285382 5976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:25:41.291361 5976 catalog_manager.cc:1347] Generated new cluster ID: 5f8d6e9082c2443cac885406a90c704c
I20260807 06:25:41.291438 5976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:25:41.300951 5976 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:25:41.302050 5976 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:25:41.314723 5976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062: Generated new TSK 0
I20260807 06:25:41.315886 5976 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
I20260807 06:25:41.356596 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:0
--local_ip_for_outbound_sockets=127.15.254.1
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:32813
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:25:41.501047 5983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:41.501325 5983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:41.501377 5983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:41.505508 5983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:41.505576 5983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:41.505635 5983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:25:41.510900 5983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:32813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:41.512038 5983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:41.513144 5983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:41.523721 5989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:41.523713 5990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:41.523722 5994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:41.524138 5983 server_base.cc:1034] running on GCE node
I20260807 06:25:41.524647 5983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:41.525369 5983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:41.526544 5983 hybrid_clock.cc:648] HybridClock initialized: now 1786083941526521 us; error 37 us; skew 500 ppm
Aug 07 06:25:41 dist-test-slave-44d2 krb5kdc[5821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083941, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:41.530073 5983 init.cc:377] Logged in from keytab as kudu/127.15.254.1@KRBTEST.COM (short username kudu)
I20260807 06:25:41.531018 5983 webserver.cc:460] Webserver started at http://127.15.254.1:40347/ using document root <none> and password file <none>
I20260807 06:25:41.531627 5983 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:41.531724 5983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:25:41.531971 5983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:25:41.534210 5983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/data/instance:
uuid: "c5faea82dbbb41fd9043ece2d9b5db40"
format_stamp: "Formatted at 2026-08-07 06:25:41 on dist-test-slave-44d2"
server_key: "1f5c61ded5a64b6fe6b2de4c7482723b"
server_key_iv: "07b653ae77c5b67a8f8b92184e413862"
server_key_version: "encryptionkey@0"
I20260807 06:25:41.534807 5983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance:
uuid: "c5faea82dbbb41fd9043ece2d9b5db40"
format_stamp: "Formatted at 2026-08-07 06:25:41 on dist-test-slave-44d2"
server_key: "1f5c61ded5a64b6fe6b2de4c7482723b"
server_key_iv: "07b653ae77c5b67a8f8b92184e413862"
server_key_version: "encryptionkey@0"
I20260807 06:25:41.538769 5983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:25:41.541347 6003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:41.542941 5983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:25:41.543071 5983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "c5faea82dbbb41fd9043ece2d9b5db40"
format_stamp: "Formatted at 2026-08-07 06:25:41 on dist-test-slave-44d2"
server_key: "1f5c61ded5a64b6fe6b2de4c7482723b"
server_key_iv: "07b653ae77c5b67a8f8b92184e413862"
server_key_version: "encryptionkey@0"
I20260807 06:25:41.543190 5983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:41.557062 5983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:41.560050 5983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:41.560252 5983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:41.560899 5983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:25:41.561725 5983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:25:41.561796 5983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:41.561877 5983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:25:41.561913 5983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:41.572762 5983 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:46811
I20260807 06:25:41.572790 6167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:46811 every 8 connection(s)
I20260807 06:25:41.573642 5983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:25:41.575851 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 5983
I20260807 06:25:41.576066 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance
I20260807 06:25:41.576426 16376 external_mini_cluster.cc:1428] Setting key 35764bf4ff8c6145cc98f4665ea85811
Aug 07 06:25:41 dist-test-slave-44d2 krb5kdc[5821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083941, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.254.2@KRBTEST.COM; defaulting to no policy
I20260807 06:25:41.589494 5948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:41.575495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:37639 (local address 127.15.254.62:32813)
0807 06:25:41.575664 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:41.575668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:41.576728 (+ 1060us) server_negotiation.cc:407] Connection header received
0807 06:25:41.578339 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:41.578343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:41.578400 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:41.578509 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:41.579833 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:41.580471 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:41.581814 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:41.582223 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:41.582348 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:41.585036 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:41.585042 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:41.585047 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:41.585085 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:41.587011 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:41.587626 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:41.587629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:41.587632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:41.587731 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:41.588107 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:41.588113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:41.588116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:41.588381 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:41.588532 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:41.589186 (+ 654us) server_negotiation.cc:299] Negotiation successful
0807 06:25:41.589329 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 06:25:41.590452 6175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:41.575955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.1:37639)
0807 06:25:41.576556 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:41.576620 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:41.578103 (+ 1483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:41.578728 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:41.578736 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:41.579110 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:41.579678 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:41.579705 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:41.580665 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:41.580669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:41.581679 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:41.581690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:41.582393 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:41.582396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:41.582573 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:41.583260 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:41.583278 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:41.584835 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:41.587155 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:41.587161 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:41.587174 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:41.587501 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:41.587849 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:41.587852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:41.587853 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:41.588001 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:41.588568 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:41.588578 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:41.588929 (+ 351us) client_negotiation.cc:769] Sending connection context
0807 06:25:41.589122 (+ 193us) client_negotiation.cc:240] Negotiation successful
0807 06:25:41.589437 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":158,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.2@KRBTEST.COM" created.
I20260807 06:25:41.592170 6168 heartbeater.cc:347] Connected to a master server at 127.15.254.62:32813
I20260807 06:25:41.592434 6168 heartbeater.cc:464] Registering TS with master...
I20260807 06:25:41.593051 6168 heartbeater.cc:511] Master 127.15.254.62:32813 requested a full tablet report, sending...
I20260807 06:25:41.594872 5885 ts_manager.cc:194] Registered new tserver with Master: c5faea82dbbb41fd9043ece2d9b5db40 (127.15.254.1:46811)
I20260807 06:25:41.596577 5885 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.1@KRBTEST.COM'} at 127.15.254.1:37639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
I20260807 06:25:41.612735 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:0
--local_ip_for_outbound_sockets=127.15.254.2
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:32813
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:25:41.732993 6179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:41.733314 6179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:41.733399 6179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:41.737465 6179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:41.737571 6179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:41.737675 6179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:25:41.743121 6179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:32813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:41.744272 6179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:41.745395 6179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:41.755863 6184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:41.755877 6185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:41.755868 6188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:41.755997 6179 server_base.cc:1034] running on GCE node
I20260807 06:25:41.756641 6179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:41.757305 6179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:41.758538 6179 hybrid_clock.cc:648] HybridClock initialized: now 1786083941758503 us; error 48 us; skew 500 ppm
Aug 07 06:25:41 dist-test-slave-44d2 krb5kdc[5821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083941, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:41.761638 6179 init.cc:377] Logged in from keytab as kudu/127.15.254.2@KRBTEST.COM (short username kudu)
I20260807 06:25:41.762547 6179 webserver.cc:460] Webserver started at http://127.15.254.2:32917/ using document root <none> and password file <none>
I20260807 06:25:41.763078 6179 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:41.763137 6179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:25:41.763303 6179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:25:41.765470 6179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/data/instance:
uuid: "81cc94e10ad9466594b28f3d66be643e"
format_stamp: "Formatted at 2026-08-07 06:25:41 on dist-test-slave-44d2"
server_key: "2ab4b6b7b45ea2bcce8e35b9b4590a26"
server_key_iv: "13ca92a8c42ed9d0de0aae0d0438486b"
server_key_version: "encryptionkey@0"
I20260807 06:25:41.765985 6179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance:
uuid: "81cc94e10ad9466594b28f3d66be643e"
format_stamp: "Formatted at 2026-08-07 06:25:41 on dist-test-slave-44d2"
server_key: "2ab4b6b7b45ea2bcce8e35b9b4590a26"
server_key_iv: "13ca92a8c42ed9d0de0aae0d0438486b"
server_key_version: "encryptionkey@0"
I20260807 06:25:41.769712 6179 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260807 06:25:41.772217 6195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:41.773593 6179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 06:25:41.773722 6179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "81cc94e10ad9466594b28f3d66be643e"
format_stamp: "Formatted at 2026-08-07 06:25:41 on dist-test-slave-44d2"
server_key: "2ab4b6b7b45ea2bcce8e35b9b4590a26"
server_key_iv: "13ca92a8c42ed9d0de0aae0d0438486b"
server_key_version: "encryptionkey@0"
I20260807 06:25:41.773835 6179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:41.804358 6179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:41.807816 6179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:41.808041 6179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:41.808768 6179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:25:41.809643 6179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:25:41.809692 6179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:41.809762 6179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:25:41.809804 6179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:41.819964 6179 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:34579
I20260807 06:25:41.820062 6360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:34579 every 8 connection(s)
I20260807 06:25:41.821038 6179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:25:41.831218 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 6179
I20260807 06:25:41.831341 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance
I20260807 06:25:41.831668 16376 external_mini_cluster.cc:1428] Setting key 009e9c9d9e748896e4a41f939e73200c
Aug 07 06:25:41 dist-test-slave-44d2 krb5kdc[5821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083941, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:25:41.836984 5948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:41.823044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:41409 (local address 127.15.254.62:32813)
0807 06:25:41.823184 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:41.823188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:41.824068 (+ 880us) server_negotiation.cc:407] Connection header received
0807 06:25:41.825650 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:41.825653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:41.825718 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:41.825820 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:41.827323 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:41.827960 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:41.829418 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:41.829668 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:41.829766 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:41.832635 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:41.832639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:41.832641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:41.832669 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:41.834317 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:41.835090 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:41.835092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:41.835094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:41.835175 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:41.835602 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:41.835604 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:41.835605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:41.835844 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:41.835970 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:41.836540 (+ 570us) server_negotiation.cc:299] Negotiation successful
0807 06:25:41.836689 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 06:25:41.837620 6366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:41.823414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.2:41409)
0807 06:25:41.823899 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:41.823944 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:41.825395 (+ 1451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:41.826111 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:41.826126 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:41.826525 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:41.827099 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:41.827137 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:41.828107 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:41.828110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:41.829265 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:41.829282 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:41.829831 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:41.829834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:41.830059 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:41.830670 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:41.830687 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:41.832383 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:41.834448 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:41.834457 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:41.834477 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:41.834958 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:41.835268 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:41.835274 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:41.835279 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:41.835493 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:41.835990 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:41.835996 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:41.836233 (+ 237us) client_negotiation.cc:769] Sending connection context
0807 06:25:41.836392 (+ 159us) client_negotiation.cc:240] Negotiation successful
0807 06:25:41.836726 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":136,"threads_started":1}
I20260807 06:25:41.838861 6364 heartbeater.cc:347] Connected to a master server at 127.15.254.62:32813
I20260807 06:25:41.839156 6364 heartbeater.cc:464] Registering TS with master...
I20260807 06:25:41.839818 6364 heartbeater.cc:511] Master 127.15.254.62:32813 requested a full tablet report, sending...
I20260807 06:25:41.841120 5885 ts_manager.cc:194] Registered new tserver with Master: 81cc94e10ad9466594b28f3d66be643e (127.15.254.2:34579)
I20260807 06:25:41.841719 5885 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.2@KRBTEST.COM'} at 127.15.254.2:41409
WARNING: no policy specified for kudu/127.15.254.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
I20260807 06:25:41.864168 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:0
--local_ip_for_outbound_sockets=127.15.254.3
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:32813
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:25:41.980866 6372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:25:41.981237 6372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:25:41.981312 6372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:25:41.985625 6372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:25:41.985697 6372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:25:41.985790 6372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:25:41.991479 6372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:32813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:25:41.992730 6372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:25:41.993880 6372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:25:42.004369 6383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:42.004402 6384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:25:42.004359 6387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:25:42.004755 6372 server_base.cc:1034] running on GCE node
I20260807 06:25:42.005213 6372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:25:42.006011 6372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:25:42.007236 6372 hybrid_clock.cc:648] HybridClock initialized: now 1786083942007195 us; error 52 us; skew 500 ppm
Aug 07 06:25:42 dist-test-slave-44d2 krb5kdc[5821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083942, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:25:42.010597 6372 init.cc:377] Logged in from keytab as kudu/127.15.254.3@KRBTEST.COM (short username kudu)
I20260807 06:25:42.011713 6372 webserver.cc:460] Webserver started at http://127.15.254.3:45089/ using document root <none> and password file <none>
I20260807 06:25:42.012413 6372 fs_manager.cc:359] Metadata directory not provided
I20260807 06:25:42.012542 6372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:25:42.012822 6372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:25:42.015020 6372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/data/instance:
uuid: "3cff2c5374984773b23af1d98976da32"
format_stamp: "Formatted at 2026-08-07 06:25:42 on dist-test-slave-44d2"
server_key: "70dd80ba50655199051d8b001c4896d8"
server_key_iv: "c98485b2cd90f986f60d5d7e2fa18c0d"
server_key_version: "encryptionkey@0"
I20260807 06:25:42.015622 6372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance:
uuid: "3cff2c5374984773b23af1d98976da32"
format_stamp: "Formatted at 2026-08-07 06:25:42 on dist-test-slave-44d2"
server_key: "70dd80ba50655199051d8b001c4896d8"
server_key_iv: "c98485b2cd90f986f60d5d7e2fa18c0d"
server_key_version: "encryptionkey@0"
I20260807 06:25:42.019727 6372 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260807 06:25:42.022569 6397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:42.024011 6372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:25:42.024161 6372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "3cff2c5374984773b23af1d98976da32"
format_stamp: "Formatted at 2026-08-07 06:25:42 on dist-test-slave-44d2"
server_key: "70dd80ba50655199051d8b001c4896d8"
server_key_iv: "c98485b2cd90f986f60d5d7e2fa18c0d"
server_key_version: "encryptionkey@0"
I20260807 06:25:42.024307 6372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:25:42.034330 6372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:25:42.037743 6372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:25:42.037961 6372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:25:42.038568 6372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:25:42.039415 6372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:25:42.039486 6372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:42.039559 6372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:25:42.039613 6372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:25:42.049593 6372 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:34587
I20260807 06:25:42.049633 6556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:34587 every 8 connection(s)
I20260807 06:25:42.050493 6372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:25:42.052074 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 6372
I20260807 06:25:42.052266 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance
I20260807 06:25:42.052790 16376 external_mini_cluster.cc:1428] Setting key 5af7aa907a4f7bb32f37a12a3662bcf2
Aug 07 06:25:42 dist-test-slave-44d2 krb5kdc[5821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083942, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:25:42.065680 5948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:42.052777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:45767 (local address 127.15.254.62:32813)
0807 06:25:42.052958 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:42.052964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:42.054002 (+ 1038us) server_negotiation.cc:407] Connection header received
0807 06:25:42.055912 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:42.055915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:42.055962 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:42.056027 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:42.057360 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.057887 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.059116 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.059330 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.059410 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:42.062092 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:42.062095 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:42.062097 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:42.062121 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:42.063697 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:42.064220 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:42.064223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:42.064224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:42.064301 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:42.064703 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:42.064706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:42.064707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:42.064885 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:42.064960 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:42.065445 (+ 485us) server_negotiation.cc:299] Negotiation successful
0807 06:25:42.065554 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 06:25:42.066747 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:42.053213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.3:45767)
0807 06:25:42.053779 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:42.053856 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:42.055697 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:42.056182 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:42.056189 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:42.056682 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:25:42.057215 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:42.057230 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.058046 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.058049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:42.058984 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:42.058995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.059515 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.059518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:42.059687 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:42.060322 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:25:42.060339 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:25:42.061912 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:25:42.063811 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:42.063816 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:42.063828 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:42.064122 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:42.064378 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:25:42.064381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:25:42.064382 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:25:42.064607 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:25:42.064989 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:25:42.064994 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:25:42.065229 (+ 235us) client_negotiation.cc:769] Sending connection context
0807 06:25:42.065382 (+ 153us) client_negotiation.cc:240] Negotiation successful
0807 06:25:42.065710 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":152,"threads_started":1}
I20260807 06:25:42.067957 6557 heartbeater.cc:347] Connected to a master server at 127.15.254.62:32813
I20260807 06:25:42.068212 6557 heartbeater.cc:464] Registering TS with master...
I20260807 06:25:42.068779 6557 heartbeater.cc:511] Master 127.15.254.62:32813 requested a full tablet report, sending...
I20260807 06:25:42.069783 5885 ts_manager.cc:194] Registered new tserver with Master: 3cff2c5374984773b23af1d98976da32 (127.15.254.3:34587)
I20260807 06:25:42.070312 5885 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.3@KRBTEST.COM'} at 127.15.254.3:45767
I20260807 06:25:42.079721 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:25:42.090273 5948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:42.082251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51382 (local address 127.15.254.62:32813)
0807 06:25:42.082437 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:42.082442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:42.082624 (+ 182us) server_negotiation.cc:407] Connection header received
0807 06:25:42.082799 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:42.082802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:42.082854 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:42.083038 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:25:42.083784 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.084326 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.085214 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.085435 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.085549 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:42.086977 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:25:42.086981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:25:42.086983 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:25:42.087020 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:25:42.088529 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:42.088964 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:42.088969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:42.088970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:42.089056 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:25:42.089342 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:25:42.089346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:25:42.089348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:25:42.089575 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:25:42.089721 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:42.089966 (+ 245us) server_negotiation.cc:299] Negotiation successful
0807 06:25:42.090095 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260807 06:25:42.093348 5885 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:25:42.096360 5885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:25:42.111670 6582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:42.106535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50030 (local address 127.15.254.3:34587)
0807 06:25:42.107062 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:42.107079 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:42.107099 (+ 20us) server_negotiation.cc:407] Connection header received
0807 06:25:42.107166 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:42.107170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:42.107353 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:42.107512 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:25:42.107966 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.109099 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.110390 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.110949 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.111078 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:42.111235 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:42.111401 (+ 166us) server_negotiation.cc:299] Negotiation successful
0807 06:25:42.111515 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":188,"threads_started":1}
I20260807 06:25:42.111668 6578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:42.106176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:34579 (local address 127.0.0.1:52574)
0807 06:25:42.106649 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:42.106663 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:42.106762 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:42.107516 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:42.107519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:42.107538 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:25:42.107616 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:42.107622 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.108799 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.108802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:42.109990 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:42.110004 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.111125 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.111129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:42.111241 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:42.111270 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 06:25:42.111325 (+ 55us) client_negotiation.cc:240] Negotiation successful
0807 06:25:42.111477 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260807 06:25:42.112004 6575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:42.105560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59060 (local address 127.15.254.1:46811)
0807 06:25:42.106354 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:42.106363 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:42.106391 (+ 28us) server_negotiation.cc:407] Connection header received
0807 06:25:42.106473 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:42.106478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:42.106705 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:42.107258 (+ 553us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:25:42.107402 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.108576 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.110177 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.110977 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.111100 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:42.111250 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:42.111745 (+ 495us) server_negotiation.cc:299] Negotiation successful
0807 06:25:42.111847 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":113,"threads_started":1}
I20260807 06:25:42.111678 6573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:42.105410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:46811 (local address 127.0.0.1:59060)
0807 06:25:42.105820 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:42.105875 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:42.106017 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:42.106889 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:42.106896 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:42.106919 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:25:42.107082 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:42.107091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.108706 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.108713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:42.110001 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:42.110013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.111314 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.111321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:42.111410 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:42.111438 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 06:25:42.111505 (+ 67us) client_negotiation.cc:240] Negotiation successful
0807 06:25:42.111565 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":28,"thread_start_us":91,"threads_started":1}
I20260807 06:25:42.111660 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:42.105623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:34587 (local address 127.0.0.1:50030)
0807 06:25:42.105989 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:42.106002 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:42.106096 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:42.107717 (+ 1621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:42.107720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:42.107740 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:25:42.107834 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:42.107840 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.109248 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.109251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:42.110277 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:42.110284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.111079 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.111082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:42.111188 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:42.111225 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 06:25:42.111366 (+ 141us) client_negotiation.cc:240] Negotiation successful
0807 06:25:42.111505 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":50,"thread_start_us":122,"threads_started":1}
I20260807 06:25:42.112228 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:42.106454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52574 (local address 127.15.254.2:34579)
0807 06:25:42.107039 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:42.107052 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:42.107077 (+ 25us) server_negotiation.cc:407] Connection header received
0807 06:25:42.107157 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:42.107162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:42.107353 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:42.107512 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:25:42.107736 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.108626 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.110133 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.110977 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.111745 (+ 768us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:42.111892 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:42.111944 (+ 52us) server_negotiation.cc:299] Negotiation successful
0807 06:25:42.112035 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":156,"threads_started":1}
I20260807 06:25:42.115469 6470 tablet_service.cc:1467] Processing CreateTablet for tablet 1e902d851ca5420a8792dee46c0ff14a (DEFAULT_TABLE table=test-table [id=a15cc8f3229d4612bf358d41464405e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:25:42.115469 6270 tablet_service.cc:1467] Processing CreateTablet for tablet 1e902d851ca5420a8792dee46c0ff14a (DEFAULT_TABLE table=test-table [id=a15cc8f3229d4612bf358d41464405e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:25:42.116910 6270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e902d851ca5420a8792dee46c0ff14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:25:42.116910 6470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e902d851ca5420a8792dee46c0ff14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:25:42.118160 6067 tablet_service.cc:1467] Processing CreateTablet for tablet 1e902d851ca5420a8792dee46c0ff14a (DEFAULT_TABLE table=test-table [id=a15cc8f3229d4612bf358d41464405e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:25:42.119069 6067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e902d851ca5420a8792dee46c0ff14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:25:42.125167 6584 tablet_bootstrap.cc:492] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40: Bootstrap starting.
I20260807 06:25:42.125898 6586 tablet_bootstrap.cc:492] T 1e902d851ca5420a8792dee46c0ff14a P 81cc94e10ad9466594b28f3d66be643e: Bootstrap starting.
I20260807 06:25:42.126713 6585 tablet_bootstrap.cc:492] T 1e902d851ca5420a8792dee46c0ff14a P 3cff2c5374984773b23af1d98976da32: Bootstrap starting.
I20260807 06:25:42.128041 6584 tablet_bootstrap.cc:654] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40: Neither blocks nor log segments found. Creating new log.
I20260807 06:25:42.128729 6586 tablet_bootstrap.cc:654] T 1e902d851ca5420a8792dee46c0ff14a P 81cc94e10ad9466594b28f3d66be643e: Neither blocks nor log segments found. Creating new log.
I20260807 06:25:42.129124 6584 log.cc:824] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40: Log is configured to *not* fsync() on all Append() calls
I20260807 06:25:42.129314 6585 tablet_bootstrap.cc:654] T 1e902d851ca5420a8792dee46c0ff14a P 3cff2c5374984773b23af1d98976da32: Neither blocks nor log segments found. Creating new log.
I20260807 06:25:42.129647 6586 log.cc:824] T 1e902d851ca5420a8792dee46c0ff14a P 81cc94e10ad9466594b28f3d66be643e: Log is configured to *not* fsync() on all Append() calls
I20260807 06:25:42.130874 6585 log.cc:824] T 1e902d851ca5420a8792dee46c0ff14a P 3cff2c5374984773b23af1d98976da32: Log is configured to *not* fsync() on all Append() calls
I20260807 06:25:42.131500 6584 tablet_bootstrap.cc:492] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40: No bootstrap required, opened a new log
I20260807 06:25:42.131716 6584 ts_tablet_manager.cc:1397] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260807 06:25:42.131860 6586 tablet_bootstrap.cc:492] T 1e902d851ca5420a8792dee46c0ff14a P 81cc94e10ad9466594b28f3d66be643e: No bootstrap required, opened a new log
I20260807 06:25:42.132064 6586 ts_tablet_manager.cc:1397] T 1e902d851ca5420a8792dee46c0ff14a P 81cc94e10ad9466594b28f3d66be643e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 06:25:42.132929 6585 tablet_bootstrap.cc:492] T 1e902d851ca5420a8792dee46c0ff14a P 3cff2c5374984773b23af1d98976da32: No bootstrap required, opened a new log
I20260807 06:25:42.133145 6585 ts_tablet_manager.cc:1397] T 1e902d851ca5420a8792dee46c0ff14a P 3cff2c5374984773b23af1d98976da32: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260807 06:25:42.136119 6586 raft_consensus.cc:348] T 1e902d851ca5420a8792dee46c0ff14a P 81cc94e10ad9466594b28f3d66be643e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } } peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } }
I20260807 06:25:42.136427 6586 raft_consensus.cc:374] T 1e902d851ca5420a8792dee46c0ff14a P 81cc94e10ad9466594b28f3d66be643e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:25:42.136318 6584 raft_consensus.cc:348] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } } peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } }
I20260807 06:25:42.136543 6586 raft_consensus.cc:729] T 1e902d851ca5420a8792dee46c0ff14a P 81cc94e10ad9466594b28f3d66be643e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81cc94e10ad9466594b28f3d66be643e, State: Initialized, Role: FOLLOWER
I20260807 06:25:42.136623 6584 raft_consensus.cc:374] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:25:42.136687 6584 raft_consensus.cc:729] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5faea82dbbb41fd9043ece2d9b5db40, State: Initialized, Role: FOLLOWER
I20260807 06:25:42.136730 6585 raft_consensus.cc:348] T 1e902d851ca5420a8792dee46c0ff14a P 3cff2c5374984773b23af1d98976da32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } } peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } }
I20260807 06:25:42.137040 6585 raft_consensus.cc:374] T 1e902d851ca5420a8792dee46c0ff14a P 3cff2c5374984773b23af1d98976da32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:25:42.137111 6585 raft_consensus.cc:729] T 1e902d851ca5420a8792dee46c0ff14a P 3cff2c5374984773b23af1d98976da32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cff2c5374984773b23af1d98976da32, State: Initialized, Role: FOLLOWER
I20260807 06:25:42.137084 6586 consensus_queue.cc:260] T 1e902d851ca5420a8792dee46c0ff14a P 81cc94e10ad9466594b28f3d66be643e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } } peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } }
I20260807 06:25:42.137084 6584 consensus_queue.cc:260] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } } peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } }
I20260807 06:25:42.137570 6585 consensus_queue.cc:260] T 1e902d851ca5420a8792dee46c0ff14a P 3cff2c5374984773b23af1d98976da32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } } peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } }
I20260807 06:25:42.138111 6584 ts_tablet_manager.cc:1428] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260807 06:25:42.138111 6586 ts_tablet_manager.cc:1428] T 1e902d851ca5420a8792dee46c0ff14a P 81cc94e10ad9466594b28f3d66be643e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260807 06:25:42.138260 6557 heartbeater.cc:503] Master 127.15.254.62:32813 was elected leader, sending a full tablet report...
I20260807 06:25:42.138450 6168 heartbeater.cc:503] Master 127.15.254.62:32813 was elected leader, sending a full tablet report...
I20260807 06:25:42.138450 6364 heartbeater.cc:503] Master 127.15.254.62:32813 was elected leader, sending a full tablet report...
I20260807 06:25:42.138561 6585 ts_tablet_manager.cc:1428] T 1e902d851ca5420a8792dee46c0ff14a P 3cff2c5374984773b23af1d98976da32: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260807 06:25:42.202632 6593 raft_consensus.cc:482] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:25:42.202927 6593 raft_consensus.cc:504] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } } peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } }
I20260807 06:25:42.204325 6593 leader_election.cc:288] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cff2c5374984773b23af1d98976da32 (127.15.254.3:34587), 81cc94e10ad9466594b28f3d66be643e (127.15.254.2:34579)
I20260807 06:25:42.209342 6582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:42.204724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:34799 (local address 127.15.254.3:34587)
0807 06:25:42.204874 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:42.204879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:42.205118 (+ 239us) server_negotiation.cc:407] Connection header received
0807 06:25:42.205277 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:42.205281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:42.205339 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:42.205426 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:25:42.206065 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.206740 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.208078 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.208643 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.208744 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:42.208789 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:42.209105 (+ 316us) server_negotiation.cc:299] Negotiation successful
0807 06:25:42.209201 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 06:25:42.209456 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:42.204583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:34587 (local address 127.15.254.1:34799)
0807 06:25:42.204978 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:42.204996 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:42.205139 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:42.205604 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:42.205607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:42.205630 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:25:42.205834 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:42.205843 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.206873 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.206877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:42.207928 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:42.207938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.208777 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.208782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:42.208923 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:42.208960 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 06:25:42.209064 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 06:25:42.209159 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":38,"thread_start_us":205,"threads_started":1}
I20260807 06:25:42.209717 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:42.205357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:41945 (local address 127.15.254.2:34579)
0807 06:25:42.205537 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:42.205541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:42.205737 (+ 196us) server_negotiation.cc:407] Connection header received
0807 06:25:42.205862 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:42.205868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:42.205940 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:42.206041 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:25:42.206438 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.207170 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.208347 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.209108 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.209242 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:42.209301 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:42.209522 (+ 221us) server_negotiation.cc:299] Negotiation successful
0807 06:25:42.209613 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 06:25:42.210134 6598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:42.205163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:34579 (local address 127.15.254.1:41945)
0807 06:25:42.205638 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:42.205650 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 06:25:42.205747 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:25:42.206131 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:25:42.206135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:25:42.206156 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:25:42.206279 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:42.206286 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.207306 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.207309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:42.208192 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:25:42.208203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.209243 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.209247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:25:42.209344 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:42.209380 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 06:25:42.209940 (+ 560us) client_negotiation.cc:240] Negotiation successful
0807 06:25:42.210027 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":4,"thread_start_us":127,"threads_started":1}
I20260807 06:25:42.210355 6498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e902d851ca5420a8792dee46c0ff14a" candidate_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cff2c5374984773b23af1d98976da32" is_pre_election: true
I20260807 06:25:42.210752 6498 raft_consensus.cc:2454] T 1e902d851ca5420a8792dee46c0ff14a P 3cff2c5374984773b23af1d98976da32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5faea82dbbb41fd9043ece2d9b5db40 in term 0.
I20260807 06:25:42.210837 6300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e902d851ca5420a8792dee46c0ff14a" candidate_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81cc94e10ad9466594b28f3d66be643e" is_pre_election: true
I20260807 06:25:42.211158 6300 raft_consensus.cc:2454] T 1e902d851ca5420a8792dee46c0ff14a P 81cc94e10ad9466594b28f3d66be643e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5faea82dbbb41fd9043ece2d9b5db40 in term 0.
I20260807 06:25:42.211421 6005 leader_election.cc:302] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cff2c5374984773b23af1d98976da32, c5faea82dbbb41fd9043ece2d9b5db40; no voters:
I20260807 06:25:42.211740 6593 raft_consensus.cc:2781] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:25:42.211824 6593 raft_consensus.cc:482] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:25:42.211856 6593 raft_consensus.cc:3037] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:25:42.213094 6593 raft_consensus.cc:504] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } } peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } }
I20260807 06:25:42.213464 6593 leader_election.cc:288] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40 [CANDIDATE]: Term 1 election: Requested vote from peers 3cff2c5374984773b23af1d98976da32 (127.15.254.3:34587), 81cc94e10ad9466594b28f3d66be643e (127.15.254.2:34579)
I20260807 06:25:42.213989 6300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e902d851ca5420a8792dee46c0ff14a" candidate_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81cc94e10ad9466594b28f3d66be643e"
I20260807 06:25:42.213989 6498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e902d851ca5420a8792dee46c0ff14a" candidate_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cff2c5374984773b23af1d98976da32"
I20260807 06:25:42.214130 6498 raft_consensus.cc:3037] T 1e902d851ca5420a8792dee46c0ff14a P 3cff2c5374984773b23af1d98976da32 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:25:42.214130 6300 raft_consensus.cc:3037] T 1e902d851ca5420a8792dee46c0ff14a P 81cc94e10ad9466594b28f3d66be643e [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:25:42.215500 6498 raft_consensus.cc:2454] T 1e902d851ca5420a8792dee46c0ff14a P 3cff2c5374984773b23af1d98976da32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5faea82dbbb41fd9043ece2d9b5db40 in term 1.
I20260807 06:25:42.215500 6300 raft_consensus.cc:2454] T 1e902d851ca5420a8792dee46c0ff14a P 81cc94e10ad9466594b28f3d66be643e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5faea82dbbb41fd9043ece2d9b5db40 in term 1.
I20260807 06:25:42.215875 6005 leader_election.cc:302] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cff2c5374984773b23af1d98976da32, c5faea82dbbb41fd9043ece2d9b5db40; no voters:
I20260807 06:25:42.216144 6593 raft_consensus.cc:2781] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:25:42.216403 6593 raft_consensus.cc:686] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 LEADER]: Becoming Leader. State: Replica: c5faea82dbbb41fd9043ece2d9b5db40, State: Running, Role: LEADER
I20260807 06:25:42.216820 6593 consensus_queue.cc:237] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } } peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } }
I20260807 06:25:42.220557 5885 catalog_manager.cc:5495] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5faea82dbbb41fd9043ece2d9b5db40 (127.15.254.1). New cstate: current_term: 1 leader_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } health_report { overall_health: UNKNOWN } } }
I20260807 06:25:42.234769 6575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:42.231384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59066 (local address 127.15.254.1:46811)
0807 06:25:42.231538 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0807 06:25:42.231542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:25:42.231555 (+ 13us) server_negotiation.cc:407] Connection header received
0807 06:25:42.231627 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:25:42.231629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:25:42.231679 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:25:42.231768 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:25:42.231992 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.232502 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.233361 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:25:42.233572 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:25:42.233649 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:25:42.234023 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:25:42.234432 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:25:42.234525 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 06:25:42.234610 (+ 85us) server_negotiation.cc:299] Negotiation successful
0807 06:25:42.234662 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 06:25:42.241070 6498 raft_consensus.cc:1261] T 1e902d851ca5420a8792dee46c0ff14a P 3cff2c5374984773b23af1d98976da32 [term 1 FOLLOWER]: Refusing update from remote peer c5faea82dbbb41fd9043ece2d9b5db40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:25:42.241070 6300 raft_consensus.cc:1261] T 1e902d851ca5420a8792dee46c0ff14a P 81cc94e10ad9466594b28f3d66be643e [term 1 FOLLOWER]: Refusing update from remote peer c5faea82dbbb41fd9043ece2d9b5db40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:25:42.241799 6599 consensus_queue.cc:1035] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:25:42.242107 6593 consensus_queue.cc:1035] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:25:42.260433 5883 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:25:42.260663 5883 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:25:42.263592 5883 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062: Sending DeleteTablet for 3 replicas of tablet 1e902d851ca5420a8792dee46c0ff14a
I20260807 06:25:42.264631 6067 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e902d851ca5420a8792dee46c0ff14a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:25:42 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:59060
I20260807 06:25:42.264830 6470 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e902d851ca5420a8792dee46c0ff14a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:25:42 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:50030
I20260807 06:25:42.265352 6270 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e902d851ca5420a8792dee46c0ff14a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:25:42 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:52574
I20260807 06:25:42.265507 6617 tablet_replica.cc:330] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40: stopping tablet replica
I20260807 06:25:42.265829 6617 raft_consensus.cc:2229] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:25:42.265897 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 5983
I20260807 06:25:42.272778 6620 tablet_replica.cc:330] T 1e902d851ca5420a8792dee46c0ff14a P 81cc94e10ad9466594b28f3d66be643e: stopping tablet replica
I20260807 06:25:42.273154 6620 raft_consensus.cc:2229] T 1e902d851ca5420a8792dee46c0ff14a P 81cc94e10ad9466594b28f3d66be643e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:25:42.273524 6620 raft_consensus.cc:2258] T 1e902d851ca5420a8792dee46c0ff14a P 81cc94e10ad9466594b28f3d66be643e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:25:42.273957 6618 tablet_replica.cc:330] T 1e902d851ca5420a8792dee46c0ff14a P 3cff2c5374984773b23af1d98976da32: stopping tablet replica
I20260807 06:25:42.274314 6618 raft_consensus.cc:2229] T 1e902d851ca5420a8792dee46c0ff14a P 3cff2c5374984773b23af1d98976da32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:25:42.274663 6618 raft_consensus.cc:2258] T 1e902d851ca5420a8792dee46c0ff14a P 3cff2c5374984773b23af1d98976da32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:25:42.275458 6620 ts_tablet_manager.cc:1905] T 1e902d851ca5420a8792dee46c0ff14a P 81cc94e10ad9466594b28f3d66be643e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:25:42.276072 6618 ts_tablet_manager.cc:1905] T 1e902d851ca5420a8792dee46c0ff14a P 3cff2c5374984773b23af1d98976da32: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260807 06:25:42.279040 5866 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.254.1:46811 (error 108)
I20260807 06:25:42.278700 6620 ts_tablet_manager.cc:1918] T 1e902d851ca5420a8792dee46c0ff14a P 81cc94e10ad9466594b28f3d66be643e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:25:42.279222 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 6179
I20260807 06:25:42.279203 6620 log.cc:1196] T 1e902d851ca5420a8792dee46c0ff14a P 81cc94e10ad9466594b28f3d66be643e: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/1e902d851ca5420a8792dee46c0ff14a
I20260807 06:25:42.279256 6618 ts_tablet_manager.cc:1918] T 1e902d851ca5420a8792dee46c0ff14a P 3cff2c5374984773b23af1d98976da32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:25:42.279345 6618 log.cc:1196] T 1e902d851ca5420a8792dee46c0ff14a P 3cff2c5374984773b23af1d98976da32: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/1e902d851ca5420a8792dee46c0ff14a
I20260807 06:25:42.279940 6618 ts_tablet_manager.cc:1939] T 1e902d851ca5420a8792dee46c0ff14a P 3cff2c5374984773b23af1d98976da32: Deleting consensus metadata
I20260807 06:25:42.281313 5863 catalog_manager.cc:4841] TS 3cff2c5374984773b23af1d98976da32 (127.15.254.3:34587): tablet 1e902d851ca5420a8792dee46c0ff14a (table test-table [id=a15cc8f3229d4612bf358d41464405e5]) successfully deleted
I20260807 06:25:42.282600 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:42.282235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:46811 (local address 127.0.0.1:59068)
0807 06:25:42.282401 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:42.282491 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.1:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 06:25:42.282802 5866 catalog_manager.cc:4584] TS c5faea82dbbb41fd9043ece2d9b5db40 (127.15.254.1:46811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1e902d851ca5420a8792dee46c0ff14a: Network error: Client connection negotiation failed: client connection to 127.15.254.1:46811: connect: Connection refused (error 111)
I20260807 06:25:42.289417 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 6372
I20260807 06:25:42.296243 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:25:42.289649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:34579 (local address 127.0.0.1:52576)
0807 06:25:42.296041 (+ 6392us) negotiation.cc:107] Waiting for socket to connect
0807 06:25:42.296125 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.2:34579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6259}
I20260807 06:25:42.304327 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 5840
I20260807 06:26:02.315428 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:32813
--webserver_interface=127.15.254.62
--webserver_port=37901
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:32813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:26:02.435411 6624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:02.435773 6624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:02.435858 6624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:02.440058 6624 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 06:26:02.440171 6624 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 06:26:02.440198 6624 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 06:26:02.440223 6624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:26:02.440265 6624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:02.440310 6624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:02.440343 6624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:26:02.440368 6624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:26:02.446049 6624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:32813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:32813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=37901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:02.447337 6624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:02.448453 6624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:02.464006 6634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:02.464040 6633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:02.464125 6624 server_base.cc:1034] running on GCE node
W20260807 06:26:02.464035 6636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:02.464962 6624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:02.466284 6624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:02.467510 6624 hybrid_clock.cc:648] HybridClock initialized: now 1786083962467479 us; error 57 us; skew 500 ppm
Aug 07 06:26:02 dist-test-slave-44d2 krb5kdc[5821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083962, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:26:02.470742 6624 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:26:02.471818 6624 webserver.cc:460] Webserver started at http://127.15.254.62:37901/ using document root <none> and password file <none>
I20260807 06:26:02.472463 6624 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:02.472651 6624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:02.477164 6624 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260807 06:26:02.479575 6644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:02.481073 6624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:26:02.481230 6624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "7e23c9fb51f6476090bb2ddc94145062"
format_stamp: "Formatted at 2026-08-07 06:25:41 on dist-test-slave-44d2"
server_key: "011dd30a1b109ee21c771c289a0b3e39"
server_key_iv: "cec72b9b3e93328f135a40bd0edc4b9f"
server_key_version: "encryptionkey@0"
I20260807 06:26:02.481947 6624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:02.506405 6624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:02.509963 6624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:02.510192 6624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:02.518252 6624 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:32813
I20260807 06:26:02.518623 6715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:32813 every 8 connection(s)
I20260807 06:26:02.519476 6624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:26:02.524108 6716 sys_catalog.cc:263] Verifying existing consensus state
I20260807 06:26:02.524973 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 6624
I20260807 06:26:02.525561 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:46811
--local_ip_for_outbound_sockets=127.15.254.1
--tserver_master_addrs=127.15.254.62:32813
--webserver_port=40347
--webserver_interface=127.15.254.1
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:26:02.526132 6716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062: Bootstrap starting.
I20260807 06:26:02.550798 6716 log.cc:824] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:02.562343 6716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 06:26:02.562804 6716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062: Bootstrap complete.
I20260807 06:26:02.568608 6716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e23c9fb51f6476090bb2ddc94145062" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 32813 } }
I20260807 06:26:02.569123 6716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e23c9fb51f6476090bb2ddc94145062, State: Initialized, Role: FOLLOWER
I20260807 06:26:02.569689 6716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e23c9fb51f6476090bb2ddc94145062" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 32813 } }
I20260807 06:26:02.569837 6716 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:26:02.569919 6716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:26:02.570011 6716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [term 1 FOLLOWER]: Advancing to term 2
I20260807 06:26:02.572772 6716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e23c9fb51f6476090bb2ddc94145062" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 32813 } }
I20260807 06:26:02.573132 6716 leader_election.cc:302] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e23c9fb51f6476090bb2ddc94145062; no voters:
I20260807 06:26:02.573522 6716 leader_election.cc:288] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 06:26:02.573616 6721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 06:26:02.573851 6721 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [term 2 LEADER]: Becoming Leader. State: Replica: 7e23c9fb51f6476090bb2ddc94145062, State: Running, Role: LEADER
I20260807 06:26:02.574219 6721 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e23c9fb51f6476090bb2ddc94145062" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 32813 } }
I20260807 06:26:02.574826 6716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:26:02.575830 6726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e23c9fb51f6476090bb2ddc94145062. Latest consensus state: current_term: 2 leader_uuid: "7e23c9fb51f6476090bb2ddc94145062" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e23c9fb51f6476090bb2ddc94145062" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 32813 } } }
I20260807 06:26:02.575906 6722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7e23c9fb51f6476090bb2ddc94145062" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e23c9fb51f6476090bb2ddc94145062" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 32813 } } }
I20260807 06:26:02.575984 6726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [sys.catalog]: This master's current role is: LEADER
I20260807 06:26:02.576021 6722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [sys.catalog]: This master's current role is: LEADER
I20260807 06:26:02.576608 6730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:26:02.582878 6730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:26:02.583755 6730 catalog_manager.cc:1259] Loaded cluster ID: 5f8d6e9082c2443cac885406a90c704c
I20260807 06:26:02.583837 6730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:26:02.586875 6730 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:26:02.587658 6730 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062: Loaded TSK: 0
I20260807 06:26:02.604534 6730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062: Generated new TSK 1
I20260807 06:26:02.604859 6730 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 06:26:02.679591 6719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:02.679859 6719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:02.679940 6719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:02.684666 6719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:02.684732 6719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:02.684842 6719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:26:02.690945 6719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:46811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=40347
--tserver_master_addrs=127.15.254.62:32813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:02.692035 6719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:02.693176 6719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:02.703696 6756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:02.703852 6719 server_base.cc:1034] running on GCE node
W20260807 06:26:02.703717 6759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:02.703696 6757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:02.704638 6719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:02.705375 6719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:02.706591 6719 hybrid_clock.cc:648] HybridClock initialized: now 1786083962706568 us; error 35 us; skew 500 ppm
Aug 07 06:26:02 dist-test-slave-44d2 krb5kdc[5821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083962, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:26:02.709900 6719 init.cc:377] Logged in from keytab as kudu/127.15.254.1@KRBTEST.COM (short username kudu)
I20260807 06:26:02.711011 6719 webserver.cc:460] Webserver started at http://127.15.254.1:40347/ using document root <none> and password file <none>
I20260807 06:26:02.711648 6719 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:02.711771 6719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:02.716267 6719 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:26:02.719141 6771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:02.720899 6719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:26:02.721078 6719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "c5faea82dbbb41fd9043ece2d9b5db40"
format_stamp: "Formatted at 2026-08-07 06:25:41 on dist-test-slave-44d2"
server_key: "1f5c61ded5a64b6fe6b2de4c7482723b"
server_key_iv: "07b653ae77c5b67a8f8b92184e413862"
server_key_version: "encryptionkey@0"
I20260807 06:26:02.721766 6719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:02.744899 6719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:02.748592 6719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:02.748821 6719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:02.749497 6719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:26:02.750749 6780 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260807 06:26:02.755214 6719 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 06:26:02.755321 6719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260807 06:26:02.755468 6719 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 06:26:02.757429 6719 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 06:26:02.757506 6719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260807 06:26:02.757804 6780 tablet_bootstrap.cc:492] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40: Bootstrap starting.
I20260807 06:26:02.768961 6719 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:46811
I20260807 06:26:02.769038 6933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:46811 every 8 connection(s)
I20260807 06:26:02.770031 6719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:26:02.776022 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 6719
I20260807 06:26:02.776615 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:34579
--local_ip_for_outbound_sockets=127.15.254.2
--tserver_master_addrs=127.15.254.62:32813
--webserver_port=32917
--webserver_interface=127.15.254.2
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:26:02.779805 6780 log.cc:824] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40: Log is configured to *not* fsync() on all Append() calls
Aug 07 06:26:02 dist-test-slave-44d2 krb5kdc[5821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083962, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:26:02.790009 6780 tablet_bootstrap.cc:492] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 06:26:02.790700 6780 tablet_bootstrap.cc:492] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40: Bootstrap complete.
I20260807 06:26:02.790735 6937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:02.772220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:49541 (local address 127.15.254.62:32813)
0807 06:26:02.772629 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:02.772654 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:02.773054 (+ 400us) server_negotiation.cc:407] Connection header received
0807 06:26:02.774202 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:02.774214 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:02.774513 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:02.774843 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:02.776178 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:02.777539 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:02.778841 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:02.779574 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:02.779766 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:02.783389 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:02.783399 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:02.783420 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:02.783460 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:02.785861 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:02.787020 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:02.787028 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:02.787034 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:02.787245 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:02.787744 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:02.787748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:02.787750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:02.788333 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:02.788529 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:02.789286 (+ 757us) server_negotiation.cc:299] Negotiation successful
0807 06:26:02.789653 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":169,"threads_started":1}
I20260807 06:26:02.790836 6939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:02.772278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.1:49541)
0807 06:26:02.772872 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:02.772919 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:02.773757 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:02.774907 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:02.774915 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:02.775427 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:26:02.775997 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:02.776014 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:02.777722 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:02.777725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:02.778687 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:02.778702 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:02.779744 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:02.779750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:02.779995 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:02.780927 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:26:02.780952 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:26:02.783100 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:26:02.786160 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:02.786169 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:02.786186 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:02.786736 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:02.787376 (+ 640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:02.787391 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:02.787395 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:02.787611 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:02.788502 (+ 891us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:26:02.788511 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:26:02.788868 (+ 357us) client_negotiation.cc:769] Sending connection context
0807 06:26:02.789118 (+ 250us) client_negotiation.cc:240] Negotiation successful
0807 06:26:02.789635 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":228,"threads_started":1}
I20260807 06:26:02.792588 6780 ts_tablet_manager.cc:1397] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40: Time spent bootstrapping tablet: real 0.035s user 0.030s sys 0.005s
I20260807 06:26:02.794492 6934 heartbeater.cc:347] Connected to a master server at 127.15.254.62:32813
I20260807 06:26:02.794770 6934 heartbeater.cc:464] Registering TS with master...
I20260807 06:26:02.795351 6934 heartbeater.cc:511] Master 127.15.254.62:32813 requested a full tablet report, sending...
I20260807 06:26:02.795863 6780 raft_consensus.cc:348] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } } peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } }
I20260807 06:26:02.796345 6780 raft_consensus.cc:729] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5faea82dbbb41fd9043ece2d9b5db40, State: Initialized, Role: FOLLOWER
I20260807 06:26:02.796990 6780 consensus_queue.cc:260] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } } peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } }
I20260807 06:26:02.798118 6780 ts_tablet_manager.cc:1428] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260807 06:26:02.798368 6662 ts_manager.cc:194] Registered new tserver with Master: c5faea82dbbb41fd9043ece2d9b5db40 (127.15.254.1:46811)
I20260807 06:26:02.802583 6662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.1@KRBTEST.COM'} at 127.15.254.1:49541
I20260807 06:26:02.805498 6934 heartbeater.cc:503] Master 127.15.254.62:32813 was elected leader, sending a full tablet report...
Aug 07 06:26:02 dist-test-slave-44d2 krb5kdc[5821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083962, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for kudu/127.15.254.1@KRBTEST.COM
I20260807 06:26:02.814097 6944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:02.802022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38350 (local address 127.15.254.1:46811)
0807 06:26:02.803017 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:02.803029 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:02.803055 (+ 26us) server_negotiation.cc:407] Connection header received
0807 06:26:02.803120 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:02.803124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:02.803292 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:02.803457 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:02.804280 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:02.805456 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:02.806652 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:02.807035 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:02.807170 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:02.809870 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:02.809880 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:02.809892 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:02.809919 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:02.811685 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:02.812295 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:02.812304 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:02.812306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:02.812405 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:02.812816 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:02.812823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:02.812826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:02.813151 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:02.813317 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:02.813704 (+ 387us) server_negotiation.cc:299] Negotiation successful
0807 06:26:02.813895 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":861,"thread_start_us":130,"threads_started":1}
I20260807 06:26:02.814097 6946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:02.801956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:46811 (local address 127.0.0.1:38350)
0807 06:26:02.802607 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:02.802660 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:02.802822 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:02.803474 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:02.803481 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:02.803975 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:26:02.804123 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:02.804133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:02.805604 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:02.805608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:02.806514 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:02.806526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:02.807175 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:02.807180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:02.807352 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:02.808028 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:26:02.808042 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:26:02.809695 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:26:02.811821 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:02.811833 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:02.811839 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:02.812129 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:02.812610 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:02.812613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:02.812614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:02.812705 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:02.813288 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:26:02.813295 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:26:02.813437 (+ 142us) client_negotiation.cc:769] Sending connection context
0807 06:26:02.813609 (+ 172us) client_negotiation.cc:240] Negotiation successful
0807 06:26:02.813858 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":192,"threads_started":1}
I20260807 06:26:02.815335 6830 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e902d851ca5420a8792dee46c0ff14a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:25:42 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:38350
I20260807 06:26:02.816447 6949 tablet_replica.cc:330] stopping tablet replica
I20260807 06:26:02.816862 6949 raft_consensus.cc:2229] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:26:02.817157 6949 raft_consensus.cc:2258] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:26:02.818297 6949 ts_tablet_manager.cc:1905] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:26:02.822631 6949 ts_tablet_manager.cc:1918] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:26:02.822785 6949 log.cc:1196] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/wal/wals/1e902d851ca5420a8792dee46c0ff14a
I20260807 06:26:02.823140 6949 ts_tablet_manager.cc:1939] T 1e902d851ca5420a8792dee46c0ff14a P c5faea82dbbb41fd9043ece2d9b5db40: Deleting consensus metadata
I20260807 06:26:02.825277 6648 catalog_manager.cc:4841] TS c5faea82dbbb41fd9043ece2d9b5db40 (127.15.254.1:46811): tablet 1e902d851ca5420a8792dee46c0ff14a (table test-table [id=a15cc8f3229d4612bf358d41464405e5]) successfully deleted
W20260807 06:26:02.910677 6940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:02.911000 6940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:02.911046 6940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:02.915307 6940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:02.915385 6940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:02.915447 6940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:26:02.921200 6940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:34579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=32917
--tserver_master_addrs=127.15.254.62:32813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:02.922412 6940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:02.923516 6940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:02.934422 6956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:02.934434 6961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:02.934495 6958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:02.934674 6940 server_base.cc:1034] running on GCE node
I20260807 06:26:02.935199 6940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:02.935971 6940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:02.937163 6940 hybrid_clock.cc:648] HybridClock initialized: now 1786083962937145 us; error 39 us; skew 500 ppm
Aug 07 06:26:02 dist-test-slave-44d2 krb5kdc[5821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083962, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:26:02.940246 6940 init.cc:377] Logged in from keytab as kudu/127.15.254.2@KRBTEST.COM (short username kudu)
I20260807 06:26:02.941275 6940 webserver.cc:460] Webserver started at http://127.15.254.2:32917/ using document root <none> and password file <none>
I20260807 06:26:02.941871 6940 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:02.941957 6940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:02.946362 6940 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:26:02.948952 6969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:02.950475 6940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:26:02.950659 6940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "81cc94e10ad9466594b28f3d66be643e"
format_stamp: "Formatted at 2026-08-07 06:25:41 on dist-test-slave-44d2"
server_key: "2ab4b6b7b45ea2bcce8e35b9b4590a26"
server_key_iv: "13ca92a8c42ed9d0de0aae0d0438486b"
server_key_version: "encryptionkey@0"
I20260807 06:26:02.951336 6940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:02.969923 6940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:02.973412 6940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:02.973647 6940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:02.974293 6940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:26:02.975550 6977 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260807 06:26:02.979851 6977 ts_tablet_manager.cc:1852] T 1e902d851ca5420a8792dee46c0ff14a P 81cc94e10ad9466594b28f3d66be643e: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260807 06:26:02.980346 6977 ts_tablet_manager.cc:1905] T 1e902d851ca5420a8792dee46c0ff14a P 81cc94e10ad9466594b28f3d66be643e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:26:02.985132 6977 ts_tablet_manager.cc:1918] T 1e902d851ca5420a8792dee46c0ff14a P 81cc94e10ad9466594b28f3d66be643e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:26:02.985333 6977 ts_tablet_manager.cc:1939] T 1e902d851ca5420a8792dee46c0ff14a P 81cc94e10ad9466594b28f3d66be643e: Deleting consensus metadata
I20260807 06:26:02.985993 6940 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260807 06:26:02.986125 6940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s user 0.000s sys 0.000s
I20260807 06:26:02.986229 6940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:26:02.986271 6940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:02.996824 6940 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:34579
I20260807 06:26:02.996942 7128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:34579 every 8 connection(s)
I20260807 06:26:02.997816 6940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:26:03.007098 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 6940
I20260807 06:26:03.007793 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:34587
--local_ip_for_outbound_sockets=127.15.254.3
--tserver_master_addrs=127.15.254.62:32813
--webserver_port=45089
--webserver_interface=127.15.254.3
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 06:26:03 dist-test-slave-44d2 krb5kdc[5821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083962, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:26:03.014137 6937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:03.000030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:56033 (local address 127.15.254.62:32813)
0807 06:26:03.000224 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:03.000228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:03.001238 (+ 1010us) server_negotiation.cc:407] Connection header received
0807 06:26:03.002293 (+ 1055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:03.002296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:03.002354 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:03.002472 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:03.003861 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.004624 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.005946 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.006247 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.006351 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:03.009962 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:03.009965 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:03.009967 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:03.009998 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:03.011674 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:03.012295 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:03.012298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:03.012299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:03.012382 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:03.012830 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:03.012834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:03.012836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:03.013046 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:03.013165 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:03.013753 (+ 588us) server_negotiation.cc:299] Negotiation successful
0807 06:26:03.013923 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 06:26:03.015323 7131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:03.000458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.2:56033)
0807 06:26:03.001064 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:03.001112 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:03.002035 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:03.002696 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:03.002703 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:03.003122 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:26:03.003670 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:03.003685 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.004784 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.004787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:03.005790 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:03.005801 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.006439 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.006442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:03.006716 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:03.007725 (+ 1009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:26:03.007749 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:26:03.009672 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:26:03.011816 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:03.011823 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:03.011843 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:03.012148 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:03.012507 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:03.012513 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:03.012518 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:03.012722 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:03.013166 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:26:03.013175 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:26:03.013466 (+ 291us) client_negotiation.cc:769] Sending connection context
0807 06:26:03.013696 (+ 230us) client_negotiation.cc:240] Negotiation successful
0807 06:26:03.014090 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":139,"threads_started":1}
I20260807 06:26:03.016813 7129 heartbeater.cc:347] Connected to a master server at 127.15.254.62:32813
I20260807 06:26:03.017139 7129 heartbeater.cc:464] Registering TS with master...
I20260807 06:26:03.017869 7129 heartbeater.cc:511] Master 127.15.254.62:32813 requested a full tablet report, sending...
I20260807 06:26:03.019357 6662 ts_manager.cc:194] Registered new tserver with Master: 81cc94e10ad9466594b28f3d66be643e (127.15.254.2:34579)
I20260807 06:26:03.020287 6662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.2@KRBTEST.COM'} at 127.15.254.2:56033
W20260807 06:26:03.131971 7135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:03.132241 7135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:03.132289 7135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:03.136440 7135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:03.136569 7135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:03.136637 7135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:26:03.142091 7135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:34587
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=45089
--tserver_master_addrs=127.15.254.62:32813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:03.143249 7135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:03.144353 7135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:03.155352 7144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:03.155404 7140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:03.155412 7146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:03.155809 7135 server_base.cc:1034] running on GCE node
I20260807 06:26:03.156262 7135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:03.157011 7135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:03.158209 7135 hybrid_clock.cc:648] HybridClock initialized: now 1786083963158194 us; error 40 us; skew 500 ppm
Aug 07 06:26:03 dist-test-slave-44d2 krb5kdc[5821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083963, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:26:03.161154 7135 init.cc:377] Logged in from keytab as kudu/127.15.254.3@KRBTEST.COM (short username kudu)
I20260807 06:26:03.162020 7135 webserver.cc:460] Webserver started at http://127.15.254.3:45089/ using document root <none> and password file <none>
I20260807 06:26:03.162563 7135 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:03.162638 7135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:03.166908 7135 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260807 06:26:03.169209 7156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:03.170656 7135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 06:26:03.170773 7135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "3cff2c5374984773b23af1d98976da32"
format_stamp: "Formatted at 2026-08-07 06:25:42 on dist-test-slave-44d2"
server_key: "70dd80ba50655199051d8b001c4896d8"
server_key_iv: "c98485b2cd90f986f60d5d7e2fa18c0d"
server_key_version: "encryptionkey@0"
I20260807 06:26:03.171331 7135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:03.190270 7135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:03.193768 7135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:03.193936 7135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:03.194523 7135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:26:03.195447 7135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:26:03.195499 7135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:03.195554 7135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:26:03.195570 7135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:03.205953 7135 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:34587
I20260807 06:26:03.205976 7338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:34587 every 8 connection(s)
I20260807 06:26:03.207109 7135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:26:03.217188 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 7135
Aug 07 06:26:03 dist-test-slave-44d2 krb5kdc[5821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083963, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:26:03.223201 6937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:03.209251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:60555 (local address 127.15.254.62:32813)
0807 06:26:03.209421 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:03.209427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:03.210395 (+ 968us) server_negotiation.cc:407] Connection header received
0807 06:26:03.211809 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:03.211812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:03.211864 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:03.211948 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:03.213324 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.213881 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.215112 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.215379 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.215495 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:03.218497 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:03.218501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:03.218504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:03.218535 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:03.220402 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:03.221102 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:03.221106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:03.221108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:03.221206 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:03.221620 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:03.221626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:03.221628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:03.221878 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:03.221997 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:03.222791 (+ 794us) server_negotiation.cc:299] Negotiation successful
0807 06:26:03.222982 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 06:26:03.224169 7342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:03.209702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.3:60555)
0807 06:26:03.210219 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:03.210266 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:03.211586 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:03.212136 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:03.212142 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:03.212600 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:26:03.213125 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:03.213140 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.214028 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.214031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:03.214953 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:03.214963 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.215573 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.215576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:03.215729 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:03.216355 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:26:03.216372 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:26:03.218228 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:26:03.220626 (+ 2398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:03.220633 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:03.220652 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:03.220977 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:03.221319 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:03.221323 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:03.221325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:03.221507 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:03.222077 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:26:03.222083 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:26:03.222469 (+ 386us) client_negotiation.cc:769] Sending connection context
0807 06:26:03.222732 (+ 263us) client_negotiation.cc:240] Negotiation successful
0807 06:26:03.223162 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":124,"threads_started":1}
I20260807 06:26:03.225683 7340 heartbeater.cc:347] Connected to a master server at 127.15.254.62:32813
I20260807 06:26:03.226018 7340 heartbeater.cc:464] Registering TS with master...
I20260807 06:26:03.226696 7340 heartbeater.cc:511] Master 127.15.254.62:32813 requested a full tablet report, sending...
I20260807 06:26:03.227347 7345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:03.218078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47994 (local address 127.15.254.62:32813)
0807 06:26:03.218356 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:03.218360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:03.218799 (+ 439us) server_negotiation.cc:407] Connection header received
0807 06:26:03.218834 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:03.218836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:03.218875 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:03.218980 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:03.219721 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.220257 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.221177 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.221409 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.221524 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:03.223157 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:03.223161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:03.223164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:03.223190 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:03.225338 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:03.225890 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:03.225894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:03.225896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:03.225987 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:03.226290 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:03.226293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:03.226294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:03.226478 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:03.226692 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:03.226951 (+ 259us) server_negotiation.cc:299] Negotiation successful
0807 06:26:03.227098 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":106,"threads_started":1}
I20260807 06:26:03.228037 6661 ts_manager.cc:194] Registered new tserver with Master: 3cff2c5374984773b23af1d98976da32 (127.15.254.3:34587)
I20260807 06:26:03.228811 6661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.3@KRBTEST.COM'} at 127.15.254.3:60555
I20260807 06:26:03.229413 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:26:03.233673 7345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:03.230103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48004 (local address 127.15.254.62:32813)
0807 06:26:03.230237 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:03.230241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:03.230439 (+ 198us) server_negotiation.cc:407] Connection header received
0807 06:26:03.230554 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:03.230557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:03.230668 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:03.230776 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:26:03.230982 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.231611 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.232559 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.232810 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.232901 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:03.233194 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:26:03.233324 (+ 130us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0807 06:26:03.233537 (+ 213us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48004: token expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260807 06:26:03.233757 7345 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48004: token expired
W20260807 06:26:03.234154 6560 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.15.254.62:32813)
I20260807 06:26:03.242161 7345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:03.234557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48020 (local address 127.15.254.62:32813)
0807 06:26:03.234683 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:03.234686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:03.234699 (+ 13us) server_negotiation.cc:407] Connection header received
0807 06:26:03.234788 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:03.234790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:03.234825 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:03.234888 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:03.235530 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.236048 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.237025 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.237234 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.237323 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:03.238722 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:03.238725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:03.238733 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:03.238756 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:03.240235 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:03.240868 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:03.240872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:03.240873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:03.241017 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:03.241337 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:03.241340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:03.241342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:03.241544 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:03.241657 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:03.241920 (+ 263us) server_negotiation.cc:299] Negotiation successful
0807 06:26:03.242036 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 06:26:03.245012 6662 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:26:03.248382 6662 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:26:03.256762 6830 tablet_service.cc:1467] Processing CreateTablet for tablet b21d7a5c5e4d4bce99dd6159a4d461f6 (DEFAULT_TABLE table=test-table [id=6e43bcf7cf9643b2a6f7e8a3f0610ad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:26:03.257534 6830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21d7a5c5e4d4bce99dd6159a4d461f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:03.260850 6780 tablet_bootstrap.cc:492] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40: Bootstrap starting.
I20260807 06:26:03.261719 6946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:03.255607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:34587 (local address 127.0.0.1:57278)
0807 06:26:03.255744 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:03.255777 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:03.255866 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:03.257176 (+ 1310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:03.257179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:03.257203 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:03.257310 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:03.257315 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.258752 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.258755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:03.259950 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:03.259962 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.261235 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.261242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:03.261341 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:03.261399 (+ 58us) client_negotiation.cc:769] Sending connection context
0807 06:26:03.261458 (+ 59us) client_negotiation.cc:240] Negotiation successful
0807 06:26:03.261568 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260807 06:26:03.261845 6780 tablet_bootstrap.cc:654] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:03.262071 7358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:03.256632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44638 (local address 127.15.254.2:34579)
0807 06:26:03.257179 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:03.257191 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:03.257215 (+ 24us) server_negotiation.cc:407] Connection header received
0807 06:26:03.257295 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:03.257300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:03.257523 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:03.257687 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:03.257997 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.259276 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.260280 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.261096 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.261205 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:03.261525 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:03.261774 (+ 249us) server_negotiation.cc:299] Negotiation successful
0807 06:26:03.261891 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":138,"threads_started":1}
I20260807 06:26:03.262326 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:03.256019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:34579 (local address 127.0.0.1:44638)
0807 06:26:03.256701 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:03.256717 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:03.256820 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:03.257754 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:03.257759 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:03.257776 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:03.257877 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:03.257885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.259388 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.259393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:03.260143 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:03.260152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.261524 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.261527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:03.261611 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:03.261632 (+ 21us) client_negotiation.cc:769] Sending connection context
0807 06:26:03.261709 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 06:26:03.262050 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":95,"threads_started":1}
I20260807 06:26:03.262375 7354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:03.255822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57278 (local address 127.15.254.3:34587)
0807 06:26:03.256394 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:03.256408 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:03.256434 (+ 26us) server_negotiation.cc:407] Connection header received
0807 06:26:03.256614 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:03.256621 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:03.256859 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:03.257062 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:03.257440 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.258606 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.260115 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.261090 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.261914 (+ 824us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:03.262074 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:03.262129 (+ 55us) server_negotiation.cc:299] Negotiation successful
0807 06:26:03.262232 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":119,"threads_started":1}
I20260807 06:26:03.264240 6780 tablet_bootstrap.cc:492] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40: No bootstrap required, opened a new log
I20260807 06:26:03.264328 6780 ts_tablet_manager.cc:1397] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260807 06:26:03.264132 7039 tablet_service.cc:1467] Processing CreateTablet for tablet b21d7a5c5e4d4bce99dd6159a4d461f6 (DEFAULT_TABLE table=test-table [id=6e43bcf7cf9643b2a6f7e8a3f0610ad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:26:03.264833 6780 raft_consensus.cc:348] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } } peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } }
I20260807 06:26:03.264967 6780 raft_consensus.cc:374] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:03.265017 6780 raft_consensus.cc:729] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5faea82dbbb41fd9043ece2d9b5db40, State: Initialized, Role: FOLLOWER
I20260807 06:26:03.265199 6780 consensus_queue.cc:260] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } } peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } }
I20260807 06:26:03.265376 7039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21d7a5c5e4d4bce99dd6159a4d461f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:03.266050 6780 ts_tablet_manager.cc:1428] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260807 06:26:03.265668 7236 tablet_service.cc:1467] Processing CreateTablet for tablet b21d7a5c5e4d4bce99dd6159a4d461f6 (DEFAULT_TABLE table=test-table [id=6e43bcf7cf9643b2a6f7e8a3f0610ad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:26:03.266839 7236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21d7a5c5e4d4bce99dd6159a4d461f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:03.270452 6977 tablet_bootstrap.cc:492] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e: Bootstrap starting.
W20260807 06:26:03.271384 6935 tablet.cc:2369] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:26:03.273141 6977 tablet_bootstrap.cc:654] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:03.274019 6977 log.cc:824] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:03.274118 7360 tablet_bootstrap.cc:492] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 3cff2c5374984773b23af1d98976da32: Bootstrap starting.
I20260807 06:26:03.276396 6977 tablet_bootstrap.cc:492] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e: No bootstrap required, opened a new log
I20260807 06:26:03.276650 6977 ts_tablet_manager.cc:1397] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260807 06:26:03.276721 7360 tablet_bootstrap.cc:654] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 3cff2c5374984773b23af1d98976da32: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:03.277879 7360 log.cc:824] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 3cff2c5374984773b23af1d98976da32: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:03.279685 7360 tablet_bootstrap.cc:492] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 3cff2c5374984773b23af1d98976da32: No bootstrap required, opened a new log
I20260807 06:26:03.279913 7360 ts_tablet_manager.cc:1397] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 3cff2c5374984773b23af1d98976da32: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 06:26:03.279788 6977 raft_consensus.cc:348] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } } peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } }
I20260807 06:26:03.280092 6977 raft_consensus.cc:374] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:03.280151 6977 raft_consensus.cc:729] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81cc94e10ad9466594b28f3d66be643e, State: Initialized, Role: FOLLOWER
I20260807 06:26:03.280660 6977 consensus_queue.cc:260] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } } peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } }
I20260807 06:26:03.281406 7129 heartbeater.cc:503] Master 127.15.254.62:32813 was elected leader, sending a full tablet report...
I20260807 06:26:03.281802 6977 ts_tablet_manager.cc:1428] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260807 06:26:03.283818 7360 raft_consensus.cc:348] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 3cff2c5374984773b23af1d98976da32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } } peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } }
I20260807 06:26:03.284217 7360 raft_consensus.cc:374] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 3cff2c5374984773b23af1d98976da32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:03.284301 7360 raft_consensus.cc:729] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 3cff2c5374984773b23af1d98976da32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cff2c5374984773b23af1d98976da32, State: Initialized, Role: FOLLOWER
I20260807 06:26:03.284759 7360 consensus_queue.cc:260] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 3cff2c5374984773b23af1d98976da32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } } peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } }
I20260807 06:26:03.285465 7340 heartbeater.cc:503] Master 127.15.254.62:32813 was elected leader, sending a full tablet report...
I20260807 06:26:03.285853 7360 ts_tablet_manager.cc:1428] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 3cff2c5374984773b23af1d98976da32: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
W20260807 06:26:03.458290 7341 tablet.cc:2369] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 3cff2c5374984773b23af1d98976da32: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 06:26:03.499164 7130 tablet.cc:2369] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:26:03.551625 7365 raft_consensus.cc:482] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:26:03.551927 7365 raft_consensus.cc:504] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } } peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } }
I20260807 06:26:03.553234 7365 leader_election.cc:288] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cff2c5374984773b23af1d98976da32 (127.15.254.3:34587), c5faea82dbbb41fd9043ece2d9b5db40 (127.15.254.1:46811)
I20260807 06:26:03.557602 7371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:03.553655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:55165 (local address 127.15.254.1:46811)
0807 06:26:03.554070 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:03.554077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:03.554118 (+ 41us) server_negotiation.cc:407] Connection header received
0807 06:26:03.554288 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:03.554291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:03.554360 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:03.554479 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:03.554748 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.555274 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.556585 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.557034 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.557175 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:03.557356 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:03.557393 (+ 37us) server_negotiation.cc:299] Negotiation successful
0807 06:26:03.557464 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":217,"threads_started":1}
I20260807 06:26:03.557612 7368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:03.553501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:46811 (local address 127.15.254.2:55165)
0807 06:26:03.554043 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:03.554058 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:03.554176 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:03.554488 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:03.554490 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:03.554508 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:03.554636 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:03.554642 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.555398 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.555403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:03.556422 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:03.556433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.557168 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.557172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:03.557267 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:03.557291 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 06:26:03.557358 (+ 67us) client_negotiation.cc:240] Negotiation successful
0807 06:26:03.557439 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":17,"thread_start_us":68,"threads_started":1}
I20260807 06:26:03.557745 7369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:03.553499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:34587 (local address 127.15.254.2:51597)
0807 06:26:03.553845 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:03.553858 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:03.554001 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:03.554299 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:03.554301 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:03.554320 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:03.554461 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:03.554467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.555342 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.555347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:03.556484 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:03.556537 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.557219 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.557223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:03.557332 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:03.557355 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 06:26:03.557453 (+ 98us) client_negotiation.cc:240] Negotiation successful
0807 06:26:03.557537 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":62,"thread_start_us":85,"threads_started":1}
I20260807 06:26:03.557672 7354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:03.553630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:51597 (local address 127.15.254.3:34587)
0807 06:26:03.553773 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:03.553778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:03.553947 (+ 169us) server_negotiation.cc:407] Connection header received
0807 06:26:03.554146 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:03.554149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:03.554195 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:03.554284 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:03.554576 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.555230 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.556666 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.557091 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.557226 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:03.557285 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:03.557471 (+ 186us) server_negotiation.cc:299] Negotiation successful
0807 06:26:03.557541 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 06:26:03.558530 7267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21d7a5c5e4d4bce99dd6159a4d461f6" candidate_uuid: "81cc94e10ad9466594b28f3d66be643e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cff2c5374984773b23af1d98976da32" is_pre_election: true
I20260807 06:26:03.558506 6852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21d7a5c5e4d4bce99dd6159a4d461f6" candidate_uuid: "81cc94e10ad9466594b28f3d66be643e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" is_pre_election: true
I20260807 06:26:03.558851 7267 raft_consensus.cc:2454] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 3cff2c5374984773b23af1d98976da32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81cc94e10ad9466594b28f3d66be643e in term 0.
I20260807 06:26:03.558854 6852 raft_consensus.cc:2454] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81cc94e10ad9466594b28f3d66be643e in term 0.
I20260807 06:26:03.559391 6970 leader_election.cc:302] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cff2c5374984773b23af1d98976da32, 81cc94e10ad9466594b28f3d66be643e; no voters:
I20260807 06:26:03.559722 7365 raft_consensus.cc:2781] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:26:03.559831 7365 raft_consensus.cc:482] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:26:03.559877 7365 raft_consensus.cc:3037] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:03.561038 7365 raft_consensus.cc:504] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } } peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } }
I20260807 06:26:03.561391 7365 leader_election.cc:288] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e [CANDIDATE]: Term 1 election: Requested vote from peers 3cff2c5374984773b23af1d98976da32 (127.15.254.3:34587), c5faea82dbbb41fd9043ece2d9b5db40 (127.15.254.1:46811)
I20260807 06:26:03.561775 6852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21d7a5c5e4d4bce99dd6159a4d461f6" candidate_uuid: "81cc94e10ad9466594b28f3d66be643e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5faea82dbbb41fd9043ece2d9b5db40"
I20260807 06:26:03.561820 7267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21d7a5c5e4d4bce99dd6159a4d461f6" candidate_uuid: "81cc94e10ad9466594b28f3d66be643e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cff2c5374984773b23af1d98976da32"
I20260807 06:26:03.561945 6852 raft_consensus.cc:3037] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:03.561955 7267 raft_consensus.cc:3037] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 3cff2c5374984773b23af1d98976da32 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:03.563050 7267 raft_consensus.cc:2454] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 3cff2c5374984773b23af1d98976da32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81cc94e10ad9466594b28f3d66be643e in term 1.
I20260807 06:26:03.563174 6852 raft_consensus.cc:2454] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81cc94e10ad9466594b28f3d66be643e in term 1.
I20260807 06:26:03.563443 6970 leader_election.cc:302] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cff2c5374984773b23af1d98976da32, 81cc94e10ad9466594b28f3d66be643e; no voters:
I20260807 06:26:03.563627 7365 raft_consensus.cc:2781] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:26:03.563859 7365 raft_consensus.cc:686] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e [term 1 LEADER]: Becoming Leader. State: Replica: 81cc94e10ad9466594b28f3d66be643e, State: Running, Role: LEADER
I20260807 06:26:03.564242 7365 consensus_queue.cc:237] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } } peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } }
I20260807 06:26:03.567993 6662 catalog_manager.cc:5495] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e reported cstate change: term changed from 0 to 1, leader changed from <none> to 81cc94e10ad9466594b28f3d66be643e (127.15.254.2). New cstate: current_term: 1 leader_uuid: "81cc94e10ad9466594b28f3d66be643e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } health_report { overall_health: UNKNOWN } } }
I20260807 06:26:03.631911 7358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:03.628053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44650 (local address 127.15.254.2:34579)
0807 06:26:03.628212 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:03.628216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:03.628230 (+ 14us) server_negotiation.cc:407] Connection header received
0807 06:26:03.628329 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:03.628331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:03.628390 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:03.628506 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:26:03.628771 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.629312 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.630215 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:03.630447 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:03.630543 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:03.630884 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:26:03.631389 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:26:03.631498 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:03.631636 (+ 138us) server_negotiation.cc:299] Negotiation successful
0807 06:26:03.631696 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 06:26:03.638746 7267 raft_consensus.cc:1261] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 3cff2c5374984773b23af1d98976da32 [term 1 FOLLOWER]: Refusing update from remote peer 81cc94e10ad9466594b28f3d66be643e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:26:03.638741 6852 raft_consensus.cc:1261] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 FOLLOWER]: Refusing update from remote peer 81cc94e10ad9466594b28f3d66be643e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:26:03.639438 7365 consensus_queue.cc:1035] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:26:03.639674 7372 consensus_queue.cc:1035] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:26:03.646893 7376 mvcc.cc:205] Tried to move back new op lower bound from 7315799915055120384 to 7315799914762477568. Current Snapshot: MvccSnapshot[applied={T|T < 7315799915055120384}]
I20260807 06:26:03.657281 6662 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:26:03.657476 6662 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:26:03.659937 6662 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062: Sending DeleteTablet for 3 replicas of tablet b21d7a5c5e4d4bce99dd6159a4d461f6
I20260807 06:26:03.660815 7236 tablet_service.cc:1514] Processing DeleteTablet for tablet b21d7a5c5e4d4bce99dd6159a4d461f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:26:03 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:57278
I20260807 06:26:03.660924 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 6719
I20260807 06:26:03.660914 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet b21d7a5c5e4d4bce99dd6159a4d461f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:26:03 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:44638
I20260807 06:26:03.673398 7389 tablet_replica.cc:330] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e: stopping tablet replica
I20260807 06:26:03.673831 7389 raft_consensus.cc:2229] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:26:03.674448 7389 raft_consensus.cc:2258] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:26:03.676952 7389 ts_tablet_manager.cc:1905] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:26:03.680058 7389 ts_tablet_manager.cc:1918] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:26:03.680160 7389 log.cc:1196] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/b21d7a5c5e4d4bce99dd6159a4d461f6
I20260807 06:26:03.680591 7389 ts_tablet_manager.cc:1939] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 81cc94e10ad9466594b28f3d66be643e: Deleting consensus metadata
I20260807 06:26:03.682150 6645 catalog_manager.cc:4841] TS 81cc94e10ad9466594b28f3d66be643e (127.15.254.2:34579): tablet b21d7a5c5e4d4bce99dd6159a4d461f6 (table test-table [id=6e43bcf7cf9643b2a6f7e8a3f0610ad7]) successfully deleted
I20260807 06:26:03.685214 7388 tablet_replica.cc:330] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 3cff2c5374984773b23af1d98976da32: stopping tablet replica
I20260807 06:26:03.685699 7388 raft_consensus.cc:2229] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 3cff2c5374984773b23af1d98976da32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:26:03.686199 7388 raft_consensus.cc:2258] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 3cff2c5374984773b23af1d98976da32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:26:03.688712 7388 ts_tablet_manager.cc:1905] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 3cff2c5374984773b23af1d98976da32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:26:03.692348 7388 ts_tablet_manager.cc:1918] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 3cff2c5374984773b23af1d98976da32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:26:03.692466 7388 log.cc:1196] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 3cff2c5374984773b23af1d98976da32: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/b21d7a5c5e4d4bce99dd6159a4d461f6
I20260807 06:26:03.692912 7388 ts_tablet_manager.cc:1939] T b21d7a5c5e4d4bce99dd6159a4d461f6 P 3cff2c5374984773b23af1d98976da32: Deleting consensus metadata
I20260807 06:26:03.694506 6645 catalog_manager.cc:4841] TS 3cff2c5374984773b23af1d98976da32 (127.15.254.3:34587): tablet b21d7a5c5e4d4bce99dd6159a4d461f6 (table test-table [id=6e43bcf7cf9643b2a6f7e8a3f0610ad7]) successfully deleted
W20260807 06:26:03.694814 6648 connection.cc:535] client connection to 127.15.254.1:46811 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260807 06:26:03.694981 6648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 06:26:03.695423 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 6940
I20260807 06:26:03.698122 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:03.697740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:46811 (local address 127.0.0.1:38354)
0807 06:26:03.697939 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:03.698022 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.1:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260807 06:26:03.698285 6648 catalog_manager.cc:4584] TS c5faea82dbbb41fd9043ece2d9b5db40 (127.15.254.1:46811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b21d7a5c5e4d4bce99dd6159a4d461f6: Network error: Client connection negotiation failed: client connection to 127.15.254.1:46811: connect: Connection refused (error 111)
I20260807 06:26:03.710100 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 7135
I20260807 06:26:03.723852 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 6624
I20260807 06:26:23.737169 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:32813
--webserver_interface=127.15.254.62
--webserver_port=37901
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:32813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:26:23.868209 7392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:23.868584 7392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:23.868709 7392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:23.872964 7392 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 06:26:23.873039 7392 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 06:26:23.873055 7392 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 06:26:23.873070 7392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:26:23.873090 7392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:23.873109 7392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:23.873126 7392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:26:23.873142 7392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:26:23.878927 7392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:32813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:32813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=37901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:23.880179 7392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:23.881376 7392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:23.891121 7400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:23.891130 7399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:23.891216 7392 server_base.cc:1034] running on GCE node
W20260807 06:26:23.891130 7404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:23.892041 7392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:23.893384 7392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:23.894596 7392 hybrid_clock.cc:648] HybridClock initialized: now 1786083983894564 us; error 44 us; skew 500 ppm
Aug 07 06:26:23 dist-test-slave-44d2 krb5kdc[5821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083983, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:26:23.897892 7392 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:26:23.898895 7392 webserver.cc:460] Webserver started at http://127.15.254.62:37901/ using document root <none> and password file <none>
I20260807 06:26:23.899549 7392 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:23.899657 7392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:23.903908 7392 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:26:23.906505 7413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:23.907950 7392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:26:23.908164 7392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "7e23c9fb51f6476090bb2ddc94145062"
format_stamp: "Formatted at 2026-08-07 06:25:41 on dist-test-slave-44d2"
server_key: "011dd30a1b109ee21c771c289a0b3e39"
server_key_iv: "cec72b9b3e93328f135a40bd0edc4b9f"
server_key_version: "encryptionkey@0"
I20260807 06:26:23.908833 7392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:23.931370 7392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:23.934978 7392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:23.935313 7392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:23.943488 7392 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:32813
I20260807 06:26:23.943794 7485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:32813 every 8 connection(s)
I20260807 06:26:23.944715 7392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:26:23.947767 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 7392
I20260807 06:26:23.948330 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:46811
--local_ip_for_outbound_sockets=127.15.254.1
--tserver_master_addrs=127.15.254.62:32813
--webserver_port=40347
--webserver_interface=127.15.254.1
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:26:23.949836 7487 sys_catalog.cc:263] Verifying existing consensus state
I20260807 06:26:23.952471 7487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062: Bootstrap starting.
I20260807 06:26:23.972297 7487 log.cc:824] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:23.987051 7487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 06:26:23.987603 7487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062: Bootstrap complete.
I20260807 06:26:23.992326 7487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e23c9fb51f6476090bb2ddc94145062" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 32813 } }
I20260807 06:26:23.992908 7487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e23c9fb51f6476090bb2ddc94145062, State: Initialized, Role: FOLLOWER
I20260807 06:26:23.993467 7487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e23c9fb51f6476090bb2ddc94145062" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 32813 } }
I20260807 06:26:23.993598 7487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:26:23.993705 7487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:26:23.993841 7487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [term 2 FOLLOWER]: Advancing to term 3
I20260807 06:26:23.996084 7487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e23c9fb51f6476090bb2ddc94145062" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 32813 } }
I20260807 06:26:23.996536 7487 leader_election.cc:302] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e23c9fb51f6476090bb2ddc94145062; no voters:
I20260807 06:26:23.997042 7487 leader_election.cc:288] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260807 06:26:23.997145 7493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 06:26:23.997387 7493 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [term 3 LEADER]: Becoming Leader. State: Replica: 7e23c9fb51f6476090bb2ddc94145062, State: Running, Role: LEADER
I20260807 06:26:23.997795 7493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e23c9fb51f6476090bb2ddc94145062" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 32813 } }
I20260807 06:26:23.998522 7487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:26:23.999583 7495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e23c9fb51f6476090bb2ddc94145062. Latest consensus state: current_term: 3 leader_uuid: "7e23c9fb51f6476090bb2ddc94145062" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e23c9fb51f6476090bb2ddc94145062" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 32813 } } }
I20260807 06:26:23.999612 7494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7e23c9fb51f6476090bb2ddc94145062" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e23c9fb51f6476090bb2ddc94145062" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 32813 } } }
I20260807 06:26:23.999745 7495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [sys.catalog]: This master's current role is: LEADER
I20260807 06:26:23.999747 7494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062 [sys.catalog]: This master's current role is: LEADER
I20260807 06:26:24.000268 7506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:26:24.006564 7506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:26:24.007514 7506 catalog_manager.cc:1259] Loaded cluster ID: 5f8d6e9082c2443cac885406a90c704c
I20260807 06:26:24.007661 7506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:26:24.010555 7506 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:26:24.011559 7506 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062: Loaded TSK: 0
I20260807 06:26:24.011759 7506 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7e23c9fb51f6476090bb2ddc94145062: Loaded TSK: 1
F20260807 06:26:24.012806 7506 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f712d0879ad google::LogMessage::Fail() at ??:0
@ 0x7f712d08a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f712d0874de google::LogMessage::Flush() at ??:0
@ 0x7f712d088a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f712fd4b18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f712fd43f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f71379ec7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f71379c699c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f71379c7178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f7137a08df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f7137a2782b std::function<>::operator()() at ??:0
@ 0x7f71379c6b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f71379c7f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f71379c4579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f7137a08640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f712f2d93d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f712f2d9c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f712f2db3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f712f2ca512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f713090e6db start_thread at ??:0
@ 0x7f712bb8561f clone at ??:0
W20260807 06:26:24.099115 7489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:24.099385 7489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:24.099433 7489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:24.104423 7489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:24.104542 7489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:24.104624 7489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:26:24.110914 7489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:46811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=40347
--tserver_master_addrs=127.15.254.62:32813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:24.112130 7489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:24.113276 7489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:24.124886 7564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:24.124984 7563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:24.125469 7489 server_base.cc:1034] running on GCE node
W20260807 06:26:24.125547 7567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:24.126014 7489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:24.126865 7489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:24.128057 7489 hybrid_clock.cc:648] HybridClock initialized: now 1786083984128026 us; error 42 us; skew 500 ppm
Aug 07 06:26:24 dist-test-slave-44d2 krb5kdc[5821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083984, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:26:24.131184 7489 init.cc:377] Logged in from keytab as kudu/127.15.254.1@KRBTEST.COM (short username kudu)
I20260807 06:26:24.132148 7489 webserver.cc:460] Webserver started at http://127.15.254.1:40347/ using document root <none> and password file <none>
I20260807 06:26:24.132952 7489 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:24.133049 7489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:24.137931 7489 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260807 06:26:24.140331 7584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:24.141840 7489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 06:26:24.141990 7489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "c5faea82dbbb41fd9043ece2d9b5db40"
format_stamp: "Formatted at 2026-08-07 06:25:41 on dist-test-slave-44d2"
server_key: "1f5c61ded5a64b6fe6b2de4c7482723b"
server_key_iv: "07b653ae77c5b67a8f8b92184e413862"
server_key_version: "encryptionkey@0"
I20260807 06:26:24.142570 7489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:24.161316 7489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:24.165025 7489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:24.165247 7489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:24.165902 7489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:26:24.167218 7595 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260807 06:26:24.173063 7489 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 06:26:24.173166 7489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260807 06:26:24.173270 7489 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 06:26:24.175110 7489 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 06:26:24.175186 7489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260807 06:26:24.175522 7595 tablet_bootstrap.cc:492] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40: Bootstrap starting.
I20260807 06:26:24.186661 7489 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:46811
I20260807 06:26:24.186944 7746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:46811 every 8 connection(s)
I20260807 06:26:24.187867 7489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:26:24.198143 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 7489
I20260807 06:26:24.198645 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:34579
--local_ip_for_outbound_sockets=127.15.254.2
--tserver_master_addrs=127.15.254.62:32813
--webserver_port=32917
--webserver_interface=127.15.254.2
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:26:24.200064 7595 log.cc:824] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:24.208947 7595 tablet_bootstrap.cc:492] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 06:26:24.209408 7595 tablet_bootstrap.cc:492] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40: Bootstrap complete.
I20260807 06:26:24.210749 7595 ts_tablet_manager.cc:1397] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40: Time spent bootstrapping tablet: real 0.035s user 0.022s sys 0.011s
I20260807 06:26:24.213410 7595 raft_consensus.cc:348] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } } peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } }
I20260807 06:26:24.213857 7595 raft_consensus.cc:729] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5faea82dbbb41fd9043ece2d9b5db40, State: Initialized, Role: FOLLOWER
I20260807 06:26:24.214411 7595 consensus_queue.cc:260] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } } peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } }
I20260807 06:26:24.215502 7595 ts_tablet_manager.cc:1428] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
W20260807 06:26:24.307911 7750 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:26:24.190065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.1:43341)
0807 06:26:24.190584 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:24.190640 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:24.191586 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:24.307026 (+115440us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":152,"threads_started":1}
W20260807 06:26:24.308666 7747 heartbeater.cc:644] Failed to heartbeat to 127.15.254.62:32813 (0 consecutive failures): Network error: Failed to ping master at 127.15.254.62:32813: Client connection negotiation failed: client connection to 127.15.254.62:32813: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 06:26:24.309585 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:24.309245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.1:34153)
0807 06:26:24.309445 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:24.309523 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260807 06:26:24.310493 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:24.310191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.1:52341)
0807 06:26:24.310335 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:24.310434 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260807 06:26:24.310799 7747 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 06:26:24.343199 7751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:24.343510 7751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:24.343556 7751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:24.348019 7751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:24.348096 7751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:24.348207 7751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:26:24.353864 7751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:34579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=32917
--tserver_master_addrs=127.15.254.62:32813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:24.355019 7751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:24.356160 7751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:24.366911 7763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:24.366878 7766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:24.366919 7764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:24.367107 7751 server_base.cc:1034] running on GCE node
I20260807 06:26:24.367698 7751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:24.368345 7751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:24.369534 7751 hybrid_clock.cc:648] HybridClock initialized: now 1786083984369502 us; error 53 us; skew 500 ppm
Aug 07 06:26:24 dist-test-slave-44d2 krb5kdc[5821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083984, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:26:24.372606 7751 init.cc:377] Logged in from keytab as kudu/127.15.254.2@KRBTEST.COM (short username kudu)
I20260807 06:26:24.373474 7751 webserver.cc:460] Webserver started at http://127.15.254.2:32917/ using document root <none> and password file <none>
I20260807 06:26:24.374029 7751 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:24.374102 7751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:24.378232 7751 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:26:24.380597 7777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:24.382031 7751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:26:24.382201 7751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "81cc94e10ad9466594b28f3d66be643e"
format_stamp: "Formatted at 2026-08-07 06:25:41 on dist-test-slave-44d2"
server_key: "2ab4b6b7b45ea2bcce8e35b9b4590a26"
server_key_iv: "13ca92a8c42ed9d0de0aae0d0438486b"
server_key_version: "encryptionkey@0"
I20260807 06:26:24.382777 7751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:24.403690 7751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:24.407413 7751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:24.407646 7751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:24.408344 7751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:26:24.409286 7751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:26:24.409336 7751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:24.409379 7751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:26:24.409420 7751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:24.418936 7751 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:34579
I20260807 06:26:24.418965 7939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:34579 every 8 connection(s)
I20260807 06:26:24.419993 7751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:26:24.422693 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 7751
I20260807 06:26:24.423198 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:34587
--local_ip_for_outbound_sockets=127.15.254.3
--tserver_master_addrs=127.15.254.62:32813
--webserver_port=45089
--webserver_interface=127.15.254.3
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:26:24.424458 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:24.422292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.2:42383)
0807 06:26:24.422993 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:24.423117 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":156,"threads_started":1}
W20260807 06:26:24.425160 7940 heartbeater.cc:644] Failed to heartbeat to 127.15.254.62:32813 (0 consecutive failures): Network error: Failed to ping master at 127.15.254.62:32813: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
I20260807 06:26:24.425925 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:24.425664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.2:35581)
0807 06:26:24.425802 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:24.425854 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 06:26:24.426661 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:24.426392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.2:59275)
0807 06:26:24.426527 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:24.426590 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 06:26:24.426867 7940 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 06:26:24.543339 7945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:24.543622 7945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:24.543666 7945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:24.548095 7945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:24.548170 7945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:24.548231 7945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:26:24.553781 7945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:34587
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=45089
--tserver_master_addrs=127.15.254.62:32813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:24.554919 7945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:24.556003 7945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:24.566787 7953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:24.566875 7954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:24.566861 7945 server_base.cc:1034] running on GCE node
W20260807 06:26:24.566792 7957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:24.567487 7945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:24.568221 7945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:24.569454 7945 hybrid_clock.cc:648] HybridClock initialized: now 1786083984569415 us; error 48 us; skew 500 ppm
Aug 07 06:26:24 dist-test-slave-44d2 krb5kdc[5821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083984, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:26:24.572408 7945 init.cc:377] Logged in from keytab as kudu/127.15.254.3@KRBTEST.COM (short username kudu)
I20260807 06:26:24.573392 7945 webserver.cc:460] Webserver started at http://127.15.254.3:45089/ using document root <none> and password file <none>
I20260807 06:26:24.574071 7945 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:24.574175 7945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:24.578258 7945 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:26:24.580689 7969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:24.582068 7945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:26:24.582209 7945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "3cff2c5374984773b23af1d98976da32"
format_stamp: "Formatted at 2026-08-07 06:25:42 on dist-test-slave-44d2"
server_key: "70dd80ba50655199051d8b001c4896d8"
server_key_iv: "c98485b2cd90f986f60d5d7e2fa18c0d"
server_key_version: "encryptionkey@0"
I20260807 06:26:24.582782 7945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:24.600718 7945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:24.603901 7945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:24.604138 7945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:24.604861 7945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:26:24.605756 7945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:26:24.605830 7945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:24.605906 7945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:26:24.605950 7945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:24.615948 7945 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:34587
I20260807 06:26:24.615988 8117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:34587 every 8 connection(s)
I20260807 06:26:24.617093 7945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:26:24.621551 8122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:24.619594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.3:38949)
0807 06:26:24.620191 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:24.620308 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":126,"threads_started":1}
W20260807 06:26:24.622110 8120 heartbeater.cc:644] Failed to heartbeat to 127.15.254.62:32813 (0 consecutive failures): Network error: Failed to ping master at 127.15.254.62:32813: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
I20260807 06:26:24.622262 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 7945
I20260807 06:26:24.622792 8122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:24.622570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.3:58755)
0807 06:26:24.622688 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:24.622736 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260807 06:26:24.623488 8122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:24.623266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.3:59907)
0807 06:26:24.623385 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:24.623442 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 06:26:24.623665 8120 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 06:26:25.312265 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:25.311705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.1:49851)
0807 06:26:25.312085 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:25.312144 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":100,"threads_started":1}
I20260807 06:26:25.427954 8125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:25.427437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.2:57445)
0807 06:26:25.427720 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:25.427785 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":91,"threads_started":1}
I20260807 06:26:25.624814 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:25.624307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.3:39135)
0807 06:26:25.624641 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:25.624700 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":82,"threads_started":1}
I20260807 06:26:26.051929 8130 raft_consensus.cc:482] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:26:26.052220 8130 raft_consensus.cc:504] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } } peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } }
I20260807 06:26:26.053539 8130 leader_election.cc:288] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3cff2c5374984773b23af1d98976da32 (127.15.254.3:34587), 81cc94e10ad9466594b28f3d66be643e (127.15.254.2:34579)
Aug 07 06:26:26 dist-test-slave-44d2 krb5kdc[5821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083984, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for kudu/127.15.254.2@KRBTEST.COM
Aug 07 06:26:26 dist-test-slave-44d2 krb5kdc[5821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786083984, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for kudu/127.15.254.3@KRBTEST.COM
I20260807 06:26:26.068351 8133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:26.054016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:45225 (local address 127.15.254.3:34587)
0807 06:26:26.054500 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:26.054534 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:26.054588 (+ 54us) server_negotiation.cc:407] Connection header received
0807 06:26:26.055300 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:26.055306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:26.055655 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:26.055987 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:26.057322 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:26.058336 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:26.059527 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:26.060015 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:26.060202 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:26.063568 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:26.063572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:26.063585 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:26.063614 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:26.065894 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:26.066356 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:26.066361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:26.066365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:26.066478 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:26.066773 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:26.066777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:26.066778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:26.067163 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:26.067315 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:26.067888 (+ 573us) server_negotiation.cc:299] Negotiation successful
0807 06:26:26.068149 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":104,"threads_started":1}
I20260807 06:26:26.068351 8132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:26.053808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:34587 (local address 127.15.254.1:45225)
0807 06:26:26.054320 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:26.054348 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:26.054463 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:26.056019 (+ 1556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:26.056024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:26.056661 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:26:26.057140 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:26.057155 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:26.058489 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:26.058492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:26.059402 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:26.059412 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:26.060149 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:26.060153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:26.060371 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:26.061179 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:26:26.061201 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:26:26.063359 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:26:26.066020 (+ 2661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:26.066024 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:26.066027 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:26.066226 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:26.066582 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:26.066586 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:26.066588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:26.066675 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:26.067282 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:26:26.067285 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:26:26.067564 (+ 279us) client_negotiation.cc:769] Sending connection context
0807 06:26:26.067761 (+ 197us) client_negotiation.cc:240] Negotiation successful
0807 06:26:26.068124 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":100,"spinlock_wait_cycles":28672,"thread_start_us":196,"threads_started":1}
I20260807 06:26:26.068377 8134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:26.054327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:34579 (local address 127.15.254.1:45171)
0807 06:26:26.054650 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:26.054668 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:26.054780 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:26.056185 (+ 1405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:26.056188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:26.056672 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:26:26.057140 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:26.057155 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:26.058490 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:26.058495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:26.059405 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:26.059416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:26.060181 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:26.060185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:26.060377 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:26.061208 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:26:26.061221 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:26:26.062995 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:26:26.065720 (+ 2725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:26.065726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:26.065741 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:26.066033 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:26.066430 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:26.066433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:26.066435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:26.066585 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:26.067268 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:26:26.067275 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:26:26.067561 (+ 286us) client_negotiation.cc:769] Sending connection context
0807 06:26:26.067762 (+ 201us) client_negotiation.cc:240] Negotiation successful
0807 06:26:26.068216 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":327,"spinlock_wait_cycles":41472,"thread_start_us":70,"threads_started":1}
I20260807 06:26:26.068396 8135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:26.054606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:45171 (local address 127.15.254.2:34579)
0807 06:26:26.054986 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:26.054996 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:26.055026 (+ 30us) server_negotiation.cc:407] Connection header received
0807 06:26:26.055662 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:26.055667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:26.055963 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:26.056244 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:26.057329 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:26.058342 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:26.059538 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:26.060031 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:26.060208 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:26.063261 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:26.063266 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:26.063280 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:26.063310 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:26.065590 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:26.066167 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:26.066173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:26.066179 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:26.066310 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:26.066687 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:26.066691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:26.066693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:26.067140 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:26.067305 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:26.067882 (+ 577us) server_negotiation.cc:299] Negotiation successful
0807 06:26:26.068184 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":77,"threads_started":1}
W20260807 06:26:26.070912 7585 leader_election.cc:341] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 81cc94e10ad9466594b28f3d66be643e (127.15.254.2:34579): Not found: Tablet not found: b21d7a5c5e4d4bce99dd6159a4d461f6
W20260807 06:26:26.071161 7585 leader_election.cc:341] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3cff2c5374984773b23af1d98976da32 (127.15.254.3:34587): Not found: Tablet not found: b21d7a5c5e4d4bce99dd6159a4d461f6
I20260807 06:26:26.071240 7585 leader_election.cc:302] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5faea82dbbb41fd9043ece2d9b5db40; no voters: 3cff2c5374984773b23af1d98976da32, 81cc94e10ad9466594b28f3d66be643e
I20260807 06:26:26.071555 8130 raft_consensus.cc:2726] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 06:26:26.313659 8132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:26.313214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.1:54093)
0807 06:26:26.313455 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:26.313546 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260807 06:26:26.429309 8136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:26.428823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.2:50959)
0807 06:26:26.429126 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:26.429187 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":85,"threads_started":1}
I20260807 06:26:26.626300 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:26.625736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.3:35339)
0807 06:26:26.626121 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:26.626197 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":81,"threads_started":1}
I20260807 06:26:27.315294 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:27.314798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.1:48123)
0807 06:26:27.315122 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:27.315189 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":98,"threads_started":1}
I20260807 06:26:27.430760 8139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:27.430265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.2:56647)
0807 06:26:27.430594 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:27.430652 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":85,"threads_started":1}
I20260807 06:26:27.583613 8141 raft_consensus.cc:482] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:26:27.583787 8141 raft_consensus.cc:504] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } } peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } }
I20260807 06:26:27.584326 8141 leader_election.cc:288] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3cff2c5374984773b23af1d98976da32 (127.15.254.3:34587), 81cc94e10ad9466594b28f3d66be643e (127.15.254.2:34579)
W20260807 06:26:27.585258 7585 leader_election.cc:341] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3cff2c5374984773b23af1d98976da32 (127.15.254.3:34587): Not found: Tablet not found: b21d7a5c5e4d4bce99dd6159a4d461f6
W20260807 06:26:27.585407 7585 leader_election.cc:341] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 81cc94e10ad9466594b28f3d66be643e (127.15.254.2:34579): Not found: Tablet not found: b21d7a5c5e4d4bce99dd6159a4d461f6
I20260807 06:26:27.585472 7585 leader_election.cc:302] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5faea82dbbb41fd9043ece2d9b5db40; no voters: 3cff2c5374984773b23af1d98976da32, 81cc94e10ad9466594b28f3d66be643e
I20260807 06:26:27.585666 8141 raft_consensus.cc:2726] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 06:26:27.627866 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:27.627416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.3:45681)
0807 06:26:27.627710 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:27.627770 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":104,"threads_started":1}
I20260807 06:26:28.316948 8143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:28.316390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.1:38551)
0807 06:26:28.316756 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:28.316831 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":123,"threads_started":1}
I20260807 06:26:28.432219 8144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:28.431666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.2:57007)
0807 06:26:28.432010 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:28.432074 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":106,"threads_started":1}
I20260807 06:26:28.629485 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:28.629007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.3:58535)
0807 06:26:28.629309 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:28.629365 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":95,"threads_started":1}
I20260807 06:26:29.318674 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:29.318103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.1:36721)
0807 06:26:29.318490 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:29.318564 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":84,"threads_started":1}
I20260807 06:26:29.433616 8149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:29.433149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.2:54085)
0807 06:26:29.433455 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:29.433515 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":93,"threads_started":1}
I20260807 06:26:29.631206 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:29.630534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.3:49565)
0807 06:26:29.630856 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:29.631068 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":90,"threads_started":1}
I20260807 06:26:30.320147 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:30.319676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.1:38869)
0807 06:26:30.319979 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:30.320041 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":89,"threads_started":1}
I20260807 06:26:30.434954 8153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:30.434484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.2:53043)
0807 06:26:30.434782 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:30.434842 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":93,"threads_started":1}
I20260807 06:26:30.632741 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:30.632144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.3:41431)
0807 06:26:30.632466 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:30.632637 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":112,"threads_started":1}
I20260807 06:26:30.875521 8155 raft_consensus.cc:482] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:26:30.875703 8155 raft_consensus.cc:504] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } } peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } }
I20260807 06:26:30.876600 8155 leader_election.cc:288] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3cff2c5374984773b23af1d98976da32 (127.15.254.3:34587), 81cc94e10ad9466594b28f3d66be643e (127.15.254.2:34579)
W20260807 06:26:30.877461 7585 leader_election.cc:341] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3cff2c5374984773b23af1d98976da32 (127.15.254.3:34587): Not found: Tablet not found: b21d7a5c5e4d4bce99dd6159a4d461f6
W20260807 06:26:30.877597 7585 leader_election.cc:341] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 81cc94e10ad9466594b28f3d66be643e (127.15.254.2:34579): Not found: Tablet not found: b21d7a5c5e4d4bce99dd6159a4d461f6
I20260807 06:26:30.877671 7585 leader_election.cc:302] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5faea82dbbb41fd9043ece2d9b5db40; no voters: 3cff2c5374984773b23af1d98976da32, 81cc94e10ad9466594b28f3d66be643e
I20260807 06:26:30.877830 8155 raft_consensus.cc:2726] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 06:26:31.321906 8156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:31.321301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.1:47811)
0807 06:26:31.321646 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:31.321715 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":106,"threads_started":1}
I20260807 06:26:31.436307 8157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:31.435769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.2:41083)
0807 06:26:31.436145 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:31.436204 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":83,"threads_started":1}
I20260807 06:26:31.634209 8158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:31.633644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.3:32769)
0807 06:26:31.634046 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:31.634109 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":85,"threads_started":1}
I20260807 06:26:32.323509 8159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:32.322950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.1:49135)
0807 06:26:32.323317 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:32.323402 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":100,"threads_started":1}
I20260807 06:26:32.437652 8163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:32.437175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.2:35981)
0807 06:26:32.437476 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:32.437540 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":92,"threads_started":1}
I20260807 06:26:32.635639 8164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:32.635110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.3:36569)
0807 06:26:32.635465 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:32.635533 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":106,"threads_started":1}
I20260807 06:26:33.325105 8165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:33.324543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.1:42925)
0807 06:26:33.324884 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:33.324960 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":117,"threads_started":1}
I20260807 06:26:33.438867 8166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:33.438461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.2:59481)
0807 06:26:33.438714 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:33.438766 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":83,"threads_started":1}
I20260807 06:26:33.636922 8167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:33.636459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.3:33011)
0807 06:26:33.636753 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:33.636812 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":82,"threads_started":1}
I20260807 06:26:34.163945 8168 raft_consensus.cc:482] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:26:34.164130 8168 raft_consensus.cc:504] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } } peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } }
I20260807 06:26:34.164646 8168 leader_election.cc:288] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3cff2c5374984773b23af1d98976da32 (127.15.254.3:34587), 81cc94e10ad9466594b28f3d66be643e (127.15.254.2:34579)
W20260807 06:26:34.165501 7585 leader_election.cc:341] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3cff2c5374984773b23af1d98976da32 (127.15.254.3:34587): Not found: Tablet not found: b21d7a5c5e4d4bce99dd6159a4d461f6
W20260807 06:26:34.165621 7585 leader_election.cc:341] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 81cc94e10ad9466594b28f3d66be643e (127.15.254.2:34579): Not found: Tablet not found: b21d7a5c5e4d4bce99dd6159a4d461f6
I20260807 06:26:34.165661 7585 leader_election.cc:302] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5faea82dbbb41fd9043ece2d9b5db40; no voters: 3cff2c5374984773b23af1d98976da32, 81cc94e10ad9466594b28f3d66be643e
I20260807 06:26:34.165789 8168 raft_consensus.cc:2726] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 06:26:34.326736 8169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:34.326241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.1:49757)
0807 06:26:34.326563 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:34.326626 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":98,"threads_started":1}
I20260807 06:26:34.440270 8170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:34.439822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.2:55721)
0807 06:26:34.440099 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:34.440168 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":148,"threads_started":1}
I20260807 06:26:34.638443 8172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:34.637896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.3:40647)
0807 06:26:34.638283 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:34.638344 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":147,"threads_started":1}
I20260807 06:26:35.328274 8173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:35.327760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.1:43567)
0807 06:26:35.328090 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:35.328156 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":112,"threads_started":1}
I20260807 06:26:35.441710 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:35.441142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.2:52435)
0807 06:26:35.441483 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:35.441552 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":112,"threads_started":1}
I20260807 06:26:35.640041 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:35.639488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.3:59843)
0807 06:26:35.639868 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:35.639936 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":121,"threads_started":1}
I20260807 06:26:36.329838 8176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:36.329369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.1:56247)
0807 06:26:36.329673 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:36.329736 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":93,"threads_started":1}
I20260807 06:26:36.443084 8178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:36.442575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.2:36465)
0807 06:26:36.442899 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:36.442971 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":83,"threads_started":1}
I20260807 06:26:36.641482 8179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:36.641041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.3:57763)
0807 06:26:36.641317 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:36.641377 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":99,"threads_started":1}
I20260807 06:26:37.331292 8181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:37.330699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.1:49911)
0807 06:26:37.331132 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:37.331191 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":83,"threads_started":1}
I20260807 06:26:37.444463 8182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:37.443913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.2:39827)
0807 06:26:37.444275 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:37.444338 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":129,"threads_started":1}
I20260807 06:26:37.642858 8184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:37.642369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.3:35037)
0807 06:26:37.642672 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:37.642731 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":102,"threads_started":1}
I20260807 06:26:38.332824 8186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:38.332328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.1:53021)
0807 06:26:38.332656 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:38.332724 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":117,"threads_started":1}
I20260807 06:26:38.445993 8187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:38.445441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.2:38287)
0807 06:26:38.445739 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:38.445805 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":89,"threads_started":1}
I20260807 06:26:38.644156 8188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:38.643711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.3:41269)
0807 06:26:38.643992 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:38.644056 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":107,"threads_started":1}
I20260807 06:26:38.655872 8189 raft_consensus.cc:482] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:26:38.656047 8189 raft_consensus.cc:504] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cff2c5374984773b23af1d98976da32" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34587 } } peers { permanent_uuid: "81cc94e10ad9466594b28f3d66be643e" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 34579 } } peers { permanent_uuid: "c5faea82dbbb41fd9043ece2d9b5db40" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 46811 } }
I20260807 06:26:38.656546 8189 leader_election.cc:288] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3cff2c5374984773b23af1d98976da32 (127.15.254.3:34587), 81cc94e10ad9466594b28f3d66be643e (127.15.254.2:34579)
W20260807 06:26:38.657459 7585 leader_election.cc:341] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3cff2c5374984773b23af1d98976da32 (127.15.254.3:34587): Not found: Tablet not found: b21d7a5c5e4d4bce99dd6159a4d461f6
W20260807 06:26:38.657666 7585 leader_election.cc:341] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 81cc94e10ad9466594b28f3d66be643e (127.15.254.2:34579): Not found: Tablet not found: b21d7a5c5e4d4bce99dd6159a4d461f6
I20260807 06:26:38.657744 7585 leader_election.cc:302] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5faea82dbbb41fd9043ece2d9b5db40; no voters: 3cff2c5374984773b23af1d98976da32, 81cc94e10ad9466594b28f3d66be643e
I20260807 06:26:38.657897 8189 raft_consensus.cc:2726] T b21d7a5c5e4d4bce99dd6159a4d461f6 P c5faea82dbbb41fd9043ece2d9b5db40 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 06:26:39.334450 8190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:39.334001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.1:36035)
0807 06:26:39.334275 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:39.334334 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":85,"threads_started":1}
I20260807 06:26:39.447659 8192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:39.447036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.2:36679)
0807 06:26:39.447457 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:39.447529 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":97,"threads_started":1}
I20260807 06:26:39.623371 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 7489
I20260807 06:26:39.638415 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 7751
I20260807 06:26:39.645985 8193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:39.645376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:32813 (local address 127.15.254.3:60231)
0807 06:26:39.645718 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:39.645795 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.62:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":96,"threads_started":1}
I20260807 06:26:39.651350 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 7945
2026-08-07T06:26:39Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59991 ms)
[----------] 30 tests from SecurityITest (260688 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:26:39 dist-test-slave-44d2 krb5kdc[8196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:26:39 dist-test-slave-44d2 krb5kdc[8196](info): set up 2 sockets
Aug 07 06:26:39 dist-test-slave-44d2 krb5kdc[8196](info): commencing operation
krb5kdc: starting...
W20260807 06:26:40.905956 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.226s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:26:41 dist-test-slave-44d2 krb5kdc[8196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:26:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:26:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:26:41.049203 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:44409
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:38247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:44409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:26:41.172158 8214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:41.172550 8214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:41.172711 8214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:41.177328 8214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:26:41.177423 8214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:41.177467 8214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:41.177531 8214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:26:41.177577 8214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:26:41.183413 8214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:38247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:44409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:44409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:41.184609 8214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:41.185684 8214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:41.195674 8220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:41.195783 8214 server_base.cc:1034] running on GCE node
W20260807 06:26:41.195720 8221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:41.195695 8223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:41.196811 8214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:41.197994 8214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:41.199196 8214 hybrid_clock.cc:648] HybridClock initialized: now 1786084001199182 us; error 28 us; skew 500 ppm
Aug 07 06:26:41 dist-test-slave-44d2 krb5kdc[8196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084001, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:26:41.202466 8214 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:26:41.203493 8214 webserver.cc:460] Webserver started at http://127.15.254.62:36583/ using document root <none> and password file <none>
I20260807 06:26:41.204138 8214 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:41.204241 8214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:41.204561 8214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:41.207237 8214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "d5b8e25e358541f58a62353a2ce9da86"
format_stamp: "Formatted at 2026-08-07 06:26:41 on dist-test-slave-44d2"
server_key: "36f361cdcf1c8a4f6877112b251a18fe"
server_key_iv: "c1a9b39744772e5a33c099101c308f48"
server_key_version: "encryptionkey@0"
I20260807 06:26:41.207866 8214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "d5b8e25e358541f58a62353a2ce9da86"
format_stamp: "Formatted at 2026-08-07 06:26:41 on dist-test-slave-44d2"
server_key: "36f361cdcf1c8a4f6877112b251a18fe"
server_key_iv: "c1a9b39744772e5a33c099101c308f48"
server_key_version: "encryptionkey@0"
I20260807 06:26:41.211975 8214 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260807 06:26:41.214743 8232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:41.216049 8214 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260807 06:26:41.216212 8214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "d5b8e25e358541f58a62353a2ce9da86"
format_stamp: "Formatted at 2026-08-07 06:26:41 on dist-test-slave-44d2"
server_key: "36f361cdcf1c8a4f6877112b251a18fe"
server_key_iv: "c1a9b39744772e5a33c099101c308f48"
server_key_version: "encryptionkey@0"
I20260807 06:26:41.216372 8214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:41.231664 8214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:41.238045 8214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:41.238328 8214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:41.247509 8214 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:44409
I20260807 06:26:41.247565 8308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:44409 every 8 connection(s)
I20260807 06:26:41.248765 8214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:26:41.249969 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 8214
I20260807 06:26:41.250139 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:26:41.250483 16376 external_mini_cluster.cc:1428] Setting key 1cd94be7e536a065425d3b010f3032d4
I20260807 06:26:41.252873 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:41.261662 8309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5b8e25e358541f58a62353a2ce9da86: Bootstrap starting.
Aug 07 06:26:41 dist-test-slave-44d2 krb5kdc[8196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:26:41.265008 8309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5b8e25e358541f58a62353a2ce9da86: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:41.265982 8309 log.cc:824] T 00000000000000000000000000000000 P d5b8e25e358541f58a62353a2ce9da86: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:41.269837 8309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5b8e25e358541f58a62353a2ce9da86: No bootstrap required, opened a new log
I20260807 06:26:41.270198 8312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:41.252097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39908 (local address 127.15.254.62:44409)
0807 06:26:41.252614 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:41.252635 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:41.252851 (+ 216us) server_negotiation.cc:407] Connection header received
0807 06:26:41.253698 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:41.253714 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:41.254061 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:41.254417 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:41.255333 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:41.256908 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:41.258226 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:41.258901 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:41.259094 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:41.262807 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:41.262816 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:41.262840 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:41.262884 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:41.266644 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:41.267166 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:41.267172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:41.267179 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:41.267374 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:41.267778 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:41.267782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:41.267784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:41.268239 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:41.268517 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:41.268817 (+ 300us) server_negotiation.cc:299] Negotiation successful
0807 06:26:41.269177 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":159,"threads_started":1}
I20260807 06:26:41.275314 8309 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5b8e25e358541f58a62353a2ce9da86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b8e25e358541f58a62353a2ce9da86" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 44409 } }
I20260807 06:26:41.275722 8309 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5b8e25e358541f58a62353a2ce9da86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:41.275796 8309 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5b8e25e358541f58a62353a2ce9da86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b8e25e358541f58a62353a2ce9da86, State: Initialized, Role: FOLLOWER
I20260807 06:26:41.276391 8309 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5b8e25e358541f58a62353a2ce9da86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b8e25e358541f58a62353a2ce9da86" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 44409 } }
I20260807 06:26:41.276616 8309 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5b8e25e358541f58a62353a2ce9da86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:26:41.276685 8309 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5b8e25e358541f58a62353a2ce9da86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:26:41.276799 8309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5b8e25e358541f58a62353a2ce9da86 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:41.278326 8309 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5b8e25e358541f58a62353a2ce9da86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b8e25e358541f58a62353a2ce9da86" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 44409 } }
I20260807 06:26:41.278823 8309 leader_election.cc:302] T 00000000000000000000000000000000 P d5b8e25e358541f58a62353a2ce9da86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5b8e25e358541f58a62353a2ce9da86; no voters:
I20260807 06:26:41.279269 8309 leader_election.cc:288] T 00000000000000000000000000000000 P d5b8e25e358541f58a62353a2ce9da86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:26:41.279428 8316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5b8e25e358541f58a62353a2ce9da86 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:26:41.279785 8316 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5b8e25e358541f58a62353a2ce9da86 [term 1 LEADER]: Becoming Leader. State: Replica: d5b8e25e358541f58a62353a2ce9da86, State: Running, Role: LEADER
I20260807 06:26:41.280158 8316 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5b8e25e358541f58a62353a2ce9da86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b8e25e358541f58a62353a2ce9da86" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 44409 } }
I20260807 06:26:41.280855 8309 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5b8e25e358541f58a62353a2ce9da86 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:26:41.282387 8318 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5b8e25e358541f58a62353a2ce9da86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5b8e25e358541f58a62353a2ce9da86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b8e25e358541f58a62353a2ce9da86" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 44409 } } }
I20260807 06:26:41.282554 8318 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5b8e25e358541f58a62353a2ce9da86 [sys.catalog]: This master's current role is: LEADER
I20260807 06:26:41.283031 8319 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5b8e25e358541f58a62353a2ce9da86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5b8e25e358541f58a62353a2ce9da86. Latest consensus state: current_term: 1 leader_uuid: "d5b8e25e358541f58a62353a2ce9da86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b8e25e358541f58a62353a2ce9da86" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 44409 } } }
I20260807 06:26:41.283646 8319 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5b8e25e358541f58a62353a2ce9da86 [sys.catalog]: This master's current role is: LEADER
I20260807 06:26:41.283789 8335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:26:41.287212 8335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:26:41.293555 8335 catalog_manager.cc:1347] Generated new cluster ID: 6601e44e00c54569be92ab70d7ea243e
I20260807 06:26:41.293664 8335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:26:41.308455 8335 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:26:41.309612 8335 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:26:41.314248 8335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5b8e25e358541f58a62353a2ce9da86: Generated new TSK 0
I20260807 06:26:41.315126 8335 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
I20260807 06:26:41.360471 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:0
--local_ip_for_outbound_sockets=127.15.254.1
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:44409
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:38247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:26:41.481046 8348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:41.481392 8348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:41.481462 8348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:41.485754 8348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:41.485817 8348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:41.485877 8348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:26:41.491299 8348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:38247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:44409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:41.492362 8348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:41.493494 8348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:41.503798 8360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:41.503764 8355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:41.503822 8348 server_base.cc:1034] running on GCE node
W20260807 06:26:41.503762 8357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:41.505431 8348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:41.506079 8348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:41.507242 8348 hybrid_clock.cc:648] HybridClock initialized: now 1786084001507229 us; error 32 us; skew 500 ppm
Aug 07 06:26:41 dist-test-slave-44d2 krb5kdc[8196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084001, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:26:41.510372 8348 init.cc:377] Logged in from keytab as kudu/127.15.254.1@KRBTEST.COM (short username kudu)
I20260807 06:26:41.511256 8348 webserver.cc:460] Webserver started at http://127.15.254.1:36725/ using document root <none> and password file <none>
I20260807 06:26:41.511845 8348 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:41.511945 8348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:41.512238 8348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:41.514473 8348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-0/data/instance:
uuid: "2165ea4f9946458d8204b0164cb2b5c7"
format_stamp: "Formatted at 2026-08-07 06:26:41 on dist-test-slave-44d2"
server_key: "5f6e8f9092f1e8e1e264d9c2c19fab4e"
server_key_iv: "551d3f3d1920699f86c6e008493fbfe2"
server_key_version: "encryptionkey@0"
I20260807 06:26:41.515115 8348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance:
uuid: "2165ea4f9946458d8204b0164cb2b5c7"
format_stamp: "Formatted at 2026-08-07 06:26:41 on dist-test-slave-44d2"
server_key: "5f6e8f9092f1e8e1e264d9c2c19fab4e"
server_key_iv: "551d3f3d1920699f86c6e008493fbfe2"
server_key_version: "encryptionkey@0"
I20260807 06:26:41.519307 8348 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 06:26:41.522310 8370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:41.523691 8348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 06:26:41.523833 8348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "2165ea4f9946458d8204b0164cb2b5c7"
format_stamp: "Formatted at 2026-08-07 06:26:41 on dist-test-slave-44d2"
server_key: "5f6e8f9092f1e8e1e264d9c2c19fab4e"
server_key_iv: "551d3f3d1920699f86c6e008493fbfe2"
server_key_version: "encryptionkey@0"
I20260807 06:26:41.523993 8348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:41.549837 8348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:41.553452 8348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:41.553684 8348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:41.554337 8348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:26:41.555215 8348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:26:41.555286 8348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:41.555359 8348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:26:41.555410 8348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:41.565090 8348 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:36519
I20260807 06:26:41.565135 8530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:36519 every 8 connection(s)
I20260807 06:26:41.566013 8348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:26:41.569468 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 8348
I20260807 06:26:41.569618 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance
I20260807 06:26:41.570013 16376 external_mini_cluster.cc:1428] Setting key 7544a5bab8dbc2cbc84ef3e8ebb58164
Aug 07 06:26:41 dist-test-slave-44d2 krb5kdc[8196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084001, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.254.2@KRBTEST.COM; defaulting to no policy
I20260807 06:26:41.582792 8312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:41.568023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:34659 (local address 127.15.254.62:44409)
0807 06:26:41.568196 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:41.568200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:41.569253 (+ 1053us) server_negotiation.cc:407] Connection header received
0807 06:26:41.571089 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:41.571092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:41.571138 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:41.571222 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:41.573022 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:41.573552 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:41.574907 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:41.575318 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:41.575448 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:41.578139 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:41.578144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:41.578150 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:41.578187 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:41.580145 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:41.580818 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:41.580826 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:41.580830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:41.580928 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:41.581395 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:41.581401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:41.581404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:41.581695 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:41.581827 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:41.582484 (+ 657us) server_negotiation.cc:299] Negotiation successful
0807 06:26:41.582598 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 06:26:41.583647 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:41.568371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:44409 (local address 127.15.254.1:34659)
0807 06:26:41.568976 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:41.569028 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:41.570849 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:41.571441 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:41.571452 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:41.572071 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:26:41.572843 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:41.572868 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:41.573686 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:41.573692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:41.574775 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:41.574785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:41.575480 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:41.575483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:41.575636 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:41.576275 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:26:41.576291 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:26:41.577959 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:26:41.580284 (+ 2325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:41.580289 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:41.580302 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:41.580695 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:41.581094 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:41.581100 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:41.581101 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:41.581264 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:41.581828 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:26:41.581838 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:26:41.582222 (+ 384us) client_negotiation.cc:769] Sending connection context
0807 06:26:41.582391 (+ 169us) client_negotiation.cc:240] Negotiation successful
0807 06:26:41.582667 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":127,"threads_started":1}
I20260807 06:26:41.585234 8532 heartbeater.cc:347] Connected to a master server at 127.15.254.62:44409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.2@KRBTEST.COM" created.
I20260807 06:26:41.585601 8532 heartbeater.cc:464] Registering TS with master...
I20260807 06:26:41.586333 8532 heartbeater.cc:511] Master 127.15.254.62:44409 requested a full tablet report, sending...
I20260807 06:26:41.588284 8261 ts_manager.cc:194] Registered new tserver with Master: 2165ea4f9946458d8204b0164cb2b5c7 (127.15.254.1:36519)
I20260807 06:26:41.589567 8261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.1@KRBTEST.COM'} at 127.15.254.1:34659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
I20260807 06:26:41.604099 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:0
--local_ip_for_outbound_sockets=127.15.254.2
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:44409
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:38247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:26:41.723235 8537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:41.723522 8537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:41.723569 8537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:41.727795 8537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:41.727871 8537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:41.727933 8537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:26:41.733564 8537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:38247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:44409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:41.734619 8537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:41.735630 8537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:41.745684 8551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:41.745690 8547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:41.745815 8537 server_base.cc:1034] running on GCE node
W20260807 06:26:41.745697 8549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:41.746515 8537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:41.747107 8537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:41.748275 8537 hybrid_clock.cc:648] HybridClock initialized: now 1786084001748252 us; error 35 us; skew 500 ppm
Aug 07 06:26:41 dist-test-slave-44d2 krb5kdc[8196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084001, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:26:41.751196 8537 init.cc:377] Logged in from keytab as kudu/127.15.254.2@KRBTEST.COM (short username kudu)
I20260807 06:26:41.752117 8537 webserver.cc:460] Webserver started at http://127.15.254.2:34897/ using document root <none> and password file <none>
I20260807 06:26:41.752794 8537 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:41.752861 8537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:41.753079 8537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:41.755121 8537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-1/data/instance:
uuid: "c6738bbabd1d4f158acabb6eb177c156"
format_stamp: "Formatted at 2026-08-07 06:26:41 on dist-test-slave-44d2"
server_key: "c4adf30adc7c7de8357ce40a101640a0"
server_key_iv: "05ad557c05398677c29f50907fef80a2"
server_key_version: "encryptionkey@0"
I20260807 06:26:41.755694 8537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance:
uuid: "c6738bbabd1d4f158acabb6eb177c156"
format_stamp: "Formatted at 2026-08-07 06:26:41 on dist-test-slave-44d2"
server_key: "c4adf30adc7c7de8357ce40a101640a0"
server_key_iv: "05ad557c05398677c29f50907fef80a2"
server_key_version: "encryptionkey@0"
I20260807 06:26:41.759522 8537 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 06:26:41.762125 8561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:41.763338 8537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260807 06:26:41.763482 8537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "c6738bbabd1d4f158acabb6eb177c156"
format_stamp: "Formatted at 2026-08-07 06:26:41 on dist-test-slave-44d2"
server_key: "c4adf30adc7c7de8357ce40a101640a0"
server_key_iv: "05ad557c05398677c29f50907fef80a2"
server_key_version: "encryptionkey@0"
I20260807 06:26:41.763597 8537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:41.784228 8537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:41.787402 8537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:41.787634 8537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:41.788283 8537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:26:41.789207 8537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:26:41.789279 8537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:41.789346 8537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:26:41.789391 8537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:41.799350 8537 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:38973
I20260807 06:26:41.799479 8720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:38973 every 8 connection(s)
I20260807 06:26:41.800352 8537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:26:41.802156 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 8537
I20260807 06:26:41.802292 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance
I20260807 06:26:41.802573 16376 external_mini_cluster.cc:1428] Setting key ee87d920f65657c21f56ce203a3c6a8a
Aug 07 06:26:41 dist-test-slave-44d2 krb5kdc[8196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084001, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.254.3@KRBTEST.COM; defaulting to no policy
I20260807 06:26:41.817628 8312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:41.803143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:58399 (local address 127.15.254.62:44409)
0807 06:26:41.803277 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:41.803281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:41.804370 (+ 1089us) server_negotiation.cc:407] Connection header received
0807 06:26:41.805640 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:41.805645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:41.805709 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:41.805770 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:41.807790 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:41.808347 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:41.809628 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:41.809876 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:41.809984 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:41.813041 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:41.813047 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:41.813051 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:41.813091 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:41.815060 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:41.815636 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:41.815643 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:41.815645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:41.815751 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:41.816213 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:41.816218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:41.816221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:41.816548 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:41.816662 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:41.817376 (+ 714us) server_negotiation.cc:299] Negotiation successful
0807 06:26:41.817490 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 06:26:41.818527 8726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:41.802980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:44409 (local address 127.15.254.2:58399)
0807 06:26:41.804207 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:41.804268 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:41.805350 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:41.806262 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:41.806271 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:41.806875 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:26:41.807614 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:41.807637 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:41.808549 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:41.808555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:41.809491 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:41.809501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:41.810061 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:41.810068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:41.810303 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:41.810973 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:26:41.810999 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:26:41.812848 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:26:41.815200 (+ 2352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:41.815206 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:41.815220 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:41.815504 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:41.815867 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:41.815870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:41.815872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:41.816086 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:41.816662 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:26:41.816668 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:26:41.817055 (+ 387us) client_negotiation.cc:769] Sending connection context
0807 06:26:41.817254 (+ 199us) client_negotiation.cc:240] Negotiation successful
0807 06:26:41.817549 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"thread_start_us":740,"threads_started":1}
I20260807 06:26:41.820093 8723 heartbeater.cc:347] Connected to a master server at 127.15.254.62:44409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.3@KRBTEST.COM" created.
I20260807 06:26:41.820441 8723 heartbeater.cc:464] Registering TS with master...
I20260807 06:26:41.821332 8723 heartbeater.cc:511] Master 127.15.254.62:44409 requested a full tablet report, sending...
I20260807 06:26:41.822651 8261 ts_manager.cc:194] Registered new tserver with Master: c6738bbabd1d4f158acabb6eb177c156 (127.15.254.2:38973)
I20260807 06:26:41.823550 8261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.2@KRBTEST.COM'} at 127.15.254.2:58399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
I20260807 06:26:41.840969 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:0
--local_ip_for_outbound_sockets=127.15.254.3
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:44409
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:38247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:26:41.966579 8729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:41.966905 8729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:41.966967 8729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:41.971316 8729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:41.971390 8729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:41.971450 8729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:26:41.976828 8729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:38247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:44409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:41.977933 8729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:41.979087 8729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:41.989977 8735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:41.990010 8738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:41.989987 8734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:41.990248 8729 server_base.cc:1034] running on GCE node
I20260807 06:26:41.990676 8729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:41.991506 8729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:41.992692 8729 hybrid_clock.cc:648] HybridClock initialized: now 1786084001992659 us; error 43 us; skew 500 ppm
Aug 07 06:26:41 dist-test-slave-44d2 krb5kdc[8196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084001, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:26:41.995713 8729 init.cc:377] Logged in from keytab as kudu/127.15.254.3@KRBTEST.COM (short username kudu)
I20260807 06:26:41.996819 8729 webserver.cc:460] Webserver started at http://127.15.254.3:44415/ using document root <none> and password file <none>
I20260807 06:26:41.997435 8729 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:41.997535 8729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:41.997792 8729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:42.000010 8729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-2/data/instance:
uuid: "a073fab4b2764065b0583d04312bfc4d"
format_stamp: "Formatted at 2026-08-07 06:26:41 on dist-test-slave-44d2"
server_key: "15d268fab7907808b5f28fd5f6b83f65"
server_key_iv: "c59919b954fdbdb0baf649f72213ce42"
server_key_version: "encryptionkey@0"
I20260807 06:26:42.000672 8729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance:
uuid: "a073fab4b2764065b0583d04312bfc4d"
format_stamp: "Formatted at 2026-08-07 06:26:41 on dist-test-slave-44d2"
server_key: "15d268fab7907808b5f28fd5f6b83f65"
server_key_iv: "c59919b954fdbdb0baf649f72213ce42"
server_key_version: "encryptionkey@0"
I20260807 06:26:42.004877 8729 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 06:26:42.007889 8746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:42.009479 8729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:26:42.009656 8729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "a073fab4b2764065b0583d04312bfc4d"
format_stamp: "Formatted at 2026-08-07 06:26:41 on dist-test-slave-44d2"
server_key: "15d268fab7907808b5f28fd5f6b83f65"
server_key_iv: "c59919b954fdbdb0baf649f72213ce42"
server_key_version: "encryptionkey@0"
I20260807 06:26:42.009799 8729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:42.038699 8729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:42.042244 8729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:42.042500 8729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:42.043169 8729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:26:42.044070 8729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:26:42.044145 8729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:42.044224 8729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:26:42.044274 8729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:42.054060 8729 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:33149
I20260807 06:26:42.054106 8912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:33149 every 8 connection(s)
I20260807 06:26:42.054978 8729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:26:42.059896 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 8729
I20260807 06:26:42.060117 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance
I20260807 06:26:42.060443 16376 external_mini_cluster.cc:1428] Setting key 3ff842d09dba52229fd8a5ffdc92154f
Aug 07 06:26:42 dist-test-slave-44d2 krb5kdc[8196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084001, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:26:42.069509 8312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:42.056815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:55449 (local address 127.15.254.62:44409)
0807 06:26:42.056992 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:42.056997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:42.057841 (+ 844us) server_negotiation.cc:407] Connection header received
0807 06:26:42.059203 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:42.059206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:42.059255 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:42.059340 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:42.060708 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.061332 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.062770 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.063016 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.063126 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:42.065742 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:42.065746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:42.065748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:42.065783 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:42.067358 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:42.067880 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:42.067883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:42.067885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:42.067991 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:42.068348 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:42.068353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:42.068354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:42.068616 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:42.068725 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:42.069211 (+ 486us) server_negotiation.cc:299] Negotiation successful
0807 06:26:42.069335 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 06:26:42.070310 8915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:42.057219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:44409 (local address 127.15.254.3:55449)
0807 06:26:42.057686 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:42.057731 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:42.058969 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:42.059523 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:42.059530 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:42.059887 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:26:42.060544 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:42.060571 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.061467 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.061470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:42.062644 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:42.062653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.063181 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.063185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:42.063379 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:42.064019 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:26:42.064036 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:26:42.065578 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:26:42.067488 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:42.067494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:42.067508 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:42.067764 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:42.068096 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:42.068101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:42.068103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:42.068248 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:42.068724 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:26:42.068730 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:26:42.068973 (+ 243us) client_negotiation.cc:769] Sending connection context
0807 06:26:42.069167 (+ 194us) client_negotiation.cc:240] Negotiation successful
0807 06:26:42.069501 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":116,"threads_started":1}
I20260807 06:26:42.071465 8913 heartbeater.cc:347] Connected to a master server at 127.15.254.62:44409
I20260807 06:26:42.071694 8913 heartbeater.cc:464] Registering TS with master...
I20260807 06:26:42.072151 8913 heartbeater.cc:511] Master 127.15.254.62:44409 requested a full tablet report, sending...
I20260807 06:26:42.073254 8261 ts_manager.cc:194] Registered new tserver with Master: a073fab4b2764065b0583d04312bfc4d (127.15.254.3:33149)
I20260807 06:26:42.074014 8261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.3@KRBTEST.COM'} at 127.15.254.3:55449
I20260807 06:26:42.074705 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:26:42.085665 8312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:42.077169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39918 (local address 127.15.254.62:44409)
0807 06:26:42.077312 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:42.077317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:42.077526 (+ 209us) server_negotiation.cc:407] Connection header received
0807 06:26:42.077673 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:42.077676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:42.077718 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:42.077811 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:42.078504 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.079047 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.079915 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.080161 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.080271 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:42.082230 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:42.082234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:42.082235 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:42.082258 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:42.083810 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:42.084532 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:42.084536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:42.084537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:42.084628 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:42.084898 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:42.084900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:42.084901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:42.085117 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:42.085225 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:42.085435 (+ 210us) server_negotiation.cc:299] Negotiation successful
0807 06:26:42.085547 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 06:26:42.088801 8260 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:26:42.091988 8260 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:26:42.107376 8930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:42.102321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50126 (local address 127.15.254.2:38973)
0807 06:26:42.102691 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:42.102701 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:42.102721 (+ 20us) server_negotiation.cc:407] Connection header received
0807 06:26:42.102786 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:42.102790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:42.102941 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:42.103104 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:42.103402 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.104294 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.105765 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.106628 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.106752 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:42.106907 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:42.107102 (+ 195us) server_negotiation.cc:299] Negotiation successful
0807 06:26:42.107202 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":127,"threads_started":1}
I20260807 06:26:42.107362 8929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:42.102126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:38973 (local address 127.0.0.1:50126)
0807 06:26:42.102379 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:42.102389 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:42.102456 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:42.103092 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:42.103098 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:42.103120 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:42.103263 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:42.103273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.104443 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.104447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:42.105628 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:42.105639 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.106765 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.106769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:42.106885 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:42.106928 (+ 43us) client_negotiation.cc:769] Sending connection context
0807 06:26:42.107068 (+ 140us) client_negotiation.cc:240] Negotiation successful
0807 06:26:42.107198 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":56,"thread_start_us":123,"threads_started":1}
I20260807 06:26:42.107515 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:42.101315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:33149 (local address 127.0.0.1:51832)
0807 06:26:42.101852 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:42.101902 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:42.102046 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:42.103316 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:42.103321 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:42.103344 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:42.103447 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:42.103455 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.104868 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.104871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:42.106055 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:42.106064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.107144 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.107150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:42.107247 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:42.107270 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 06:26:42.107317 (+ 47us) client_negotiation.cc:240] Negotiation successful
0807 06:26:42.107382 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":2,"thread_start_us":74,"threads_started":1}
I20260807 06:26:42.108079 8931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:42.102479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51832 (local address 127.15.254.3:33149)
0807 06:26:42.102834 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:42.102847 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:42.102870 (+ 23us) server_negotiation.cc:407] Connection header received
0807 06:26:42.102954 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:42.102960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:42.103160 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:42.103335 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:42.103556 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.104703 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.106187 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.107002 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.107625 (+ 623us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:42.107767 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:42.107819 (+ 52us) server_negotiation.cc:299] Negotiation successful
0807 06:26:42.107907 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":168,"threads_started":1}
I20260807 06:26:42.108548 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:42.101480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:36519 (local address 127.0.0.1:37278)
0807 06:26:42.101981 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:42.101993 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:42.102095 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:42.104231 (+ 2136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:42.104234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:42.104253 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:42.104368 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:42.104378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.105944 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.105947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:42.107095 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:42.107108 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.108161 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.108168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:42.108262 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:42.108286 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 06:26:42.108343 (+ 57us) client_negotiation.cc:240] Negotiation successful
0807 06:26:42.108406 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":105,"threads_started":1}
I20260807 06:26:42.109172 8932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:42.102918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37278 (local address 127.15.254.1:36519)
0807 06:26:42.103797 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:42.103803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:42.103826 (+ 23us) server_negotiation.cc:407] Connection header received
0807 06:26:42.103913 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:42.103917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:42.104086 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:42.104231 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:42.104571 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.105822 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.107241 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.108013 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.108729 (+ 716us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:42.108876 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:42.108931 (+ 55us) server_negotiation.cc:299] Negotiation successful
0807 06:26:42.109031 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":138,"threads_started":1}
I20260807 06:26:42.110836 8629 tablet_service.cc:1467] Processing CreateTablet for tablet 17ce77e57f7f4489809676f44fd21736 (DEFAULT_TABLE table=test-table [id=5e1353fde38b44078d494200f2b5952d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:26:42.111263 8442 tablet_service.cc:1467] Processing CreateTablet for tablet 17ce77e57f7f4489809676f44fd21736 (DEFAULT_TABLE table=test-table [id=5e1353fde38b44078d494200f2b5952d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:26:42.112015 8629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ce77e57f7f4489809676f44fd21736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:42.112316 8442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ce77e57f7f4489809676f44fd21736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:42.113452 8819 tablet_service.cc:1467] Processing CreateTablet for tablet 17ce77e57f7f4489809676f44fd21736 (DEFAULT_TABLE table=test-table [id=5e1353fde38b44078d494200f2b5952d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:26:42.114552 8819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ce77e57f7f4489809676f44fd21736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:42.119189 8933 tablet_bootstrap.cc:492] T 17ce77e57f7f4489809676f44fd21736 P 2165ea4f9946458d8204b0164cb2b5c7: Bootstrap starting.
I20260807 06:26:42.120879 8935 tablet_bootstrap.cc:492] T 17ce77e57f7f4489809676f44fd21736 P c6738bbabd1d4f158acabb6eb177c156: Bootstrap starting.
I20260807 06:26:42.121868 8933 tablet_bootstrap.cc:654] T 17ce77e57f7f4489809676f44fd21736 P 2165ea4f9946458d8204b0164cb2b5c7: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:42.122815 8933 log.cc:824] T 17ce77e57f7f4489809676f44fd21736 P 2165ea4f9946458d8204b0164cb2b5c7: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:42.122939 8936 tablet_bootstrap.cc:492] T 17ce77e57f7f4489809676f44fd21736 P a073fab4b2764065b0583d04312bfc4d: Bootstrap starting.
I20260807 06:26:42.123597 8935 tablet_bootstrap.cc:654] T 17ce77e57f7f4489809676f44fd21736 P c6738bbabd1d4f158acabb6eb177c156: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:42.124524 8935 log.cc:824] T 17ce77e57f7f4489809676f44fd21736 P c6738bbabd1d4f158acabb6eb177c156: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:42.125046 8936 tablet_bootstrap.cc:654] T 17ce77e57f7f4489809676f44fd21736 P a073fab4b2764065b0583d04312bfc4d: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:42.125201 8933 tablet_bootstrap.cc:492] T 17ce77e57f7f4489809676f44fd21736 P 2165ea4f9946458d8204b0164cb2b5c7: No bootstrap required, opened a new log
I20260807 06:26:42.125430 8933 ts_tablet_manager.cc:1397] T 17ce77e57f7f4489809676f44fd21736 P 2165ea4f9946458d8204b0164cb2b5c7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260807 06:26:42.126228 8936 log.cc:824] T 17ce77e57f7f4489809676f44fd21736 P a073fab4b2764065b0583d04312bfc4d: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:42.126597 8935 tablet_bootstrap.cc:492] T 17ce77e57f7f4489809676f44fd21736 P c6738bbabd1d4f158acabb6eb177c156: No bootstrap required, opened a new log
I20260807 06:26:42.126784 8935 ts_tablet_manager.cc:1397] T 17ce77e57f7f4489809676f44fd21736 P c6738bbabd1d4f158acabb6eb177c156: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 06:26:42.128345 8936 tablet_bootstrap.cc:492] T 17ce77e57f7f4489809676f44fd21736 P a073fab4b2764065b0583d04312bfc4d: No bootstrap required, opened a new log
I20260807 06:26:42.128643 8936 ts_tablet_manager.cc:1397] T 17ce77e57f7f4489809676f44fd21736 P a073fab4b2764065b0583d04312bfc4d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260807 06:26:42.130306 8933 raft_consensus.cc:348] T 17ce77e57f7f4489809676f44fd21736 P 2165ea4f9946458d8204b0164cb2b5c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a073fab4b2764065b0583d04312bfc4d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33149 } } peers { permanent_uuid: "c6738bbabd1d4f158acabb6eb177c156" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38973 } } peers { permanent_uuid: "2165ea4f9946458d8204b0164cb2b5c7" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36519 } }
I20260807 06:26:42.130614 8933 raft_consensus.cc:374] T 17ce77e57f7f4489809676f44fd21736 P 2165ea4f9946458d8204b0164cb2b5c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:42.130686 8933 raft_consensus.cc:729] T 17ce77e57f7f4489809676f44fd21736 P 2165ea4f9946458d8204b0164cb2b5c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2165ea4f9946458d8204b0164cb2b5c7, State: Initialized, Role: FOLLOWER
I20260807 06:26:42.131155 8933 consensus_queue.cc:260] T 17ce77e57f7f4489809676f44fd21736 P 2165ea4f9946458d8204b0164cb2b5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a073fab4b2764065b0583d04312bfc4d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33149 } } peers { permanent_uuid: "c6738bbabd1d4f158acabb6eb177c156" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38973 } } peers { permanent_uuid: "2165ea4f9946458d8204b0164cb2b5c7" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36519 } }
I20260807 06:26:42.131096 8935 raft_consensus.cc:348] T 17ce77e57f7f4489809676f44fd21736 P c6738bbabd1d4f158acabb6eb177c156 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a073fab4b2764065b0583d04312bfc4d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33149 } } peers { permanent_uuid: "c6738bbabd1d4f158acabb6eb177c156" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38973 } } peers { permanent_uuid: "2165ea4f9946458d8204b0164cb2b5c7" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36519 } }
I20260807 06:26:42.131351 8935 raft_consensus.cc:374] T 17ce77e57f7f4489809676f44fd21736 P c6738bbabd1d4f158acabb6eb177c156 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:42.131412 8935 raft_consensus.cc:729] T 17ce77e57f7f4489809676f44fd21736 P c6738bbabd1d4f158acabb6eb177c156 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6738bbabd1d4f158acabb6eb177c156, State: Initialized, Role: FOLLOWER
I20260807 06:26:42.131948 8532 heartbeater.cc:503] Master 127.15.254.62:44409 was elected leader, sending a full tablet report...
I20260807 06:26:42.131896 8935 consensus_queue.cc:260] T 17ce77e57f7f4489809676f44fd21736 P c6738bbabd1d4f158acabb6eb177c156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a073fab4b2764065b0583d04312bfc4d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33149 } } peers { permanent_uuid: "c6738bbabd1d4f158acabb6eb177c156" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38973 } } peers { permanent_uuid: "2165ea4f9946458d8204b0164cb2b5c7" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36519 } }
I20260807 06:26:42.131899 8936 raft_consensus.cc:348] T 17ce77e57f7f4489809676f44fd21736 P a073fab4b2764065b0583d04312bfc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a073fab4b2764065b0583d04312bfc4d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33149 } } peers { permanent_uuid: "c6738bbabd1d4f158acabb6eb177c156" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38973 } } peers { permanent_uuid: "2165ea4f9946458d8204b0164cb2b5c7" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36519 } }
I20260807 06:26:42.132167 8936 raft_consensus.cc:374] T 17ce77e57f7f4489809676f44fd21736 P a073fab4b2764065b0583d04312bfc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:42.132226 8936 raft_consensus.cc:729] T 17ce77e57f7f4489809676f44fd21736 P a073fab4b2764065b0583d04312bfc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a073fab4b2764065b0583d04312bfc4d, State: Initialized, Role: FOLLOWER
I20260807 06:26:42.132254 8933 ts_tablet_manager.cc:1428] T 17ce77e57f7f4489809676f44fd21736 P 2165ea4f9946458d8204b0164cb2b5c7: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260807 06:26:42.132676 8936 consensus_queue.cc:260] T 17ce77e57f7f4489809676f44fd21736 P a073fab4b2764065b0583d04312bfc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a073fab4b2764065b0583d04312bfc4d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33149 } } peers { permanent_uuid: "c6738bbabd1d4f158acabb6eb177c156" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38973 } } peers { permanent_uuid: "2165ea4f9946458d8204b0164cb2b5c7" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36519 } }
I20260807 06:26:42.132874 8935 ts_tablet_manager.cc:1428] T 17ce77e57f7f4489809676f44fd21736 P c6738bbabd1d4f158acabb6eb177c156: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260807 06:26:42.132929 8723 heartbeater.cc:503] Master 127.15.254.62:44409 was elected leader, sending a full tablet report...
I20260807 06:26:42.133603 8936 ts_tablet_manager.cc:1428] T 17ce77e57f7f4489809676f44fd21736 P a073fab4b2764065b0583d04312bfc4d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260807 06:26:42.133762 8913 heartbeater.cc:503] Master 127.15.254.62:44409 was elected leader, sending a full tablet report...
I20260807 06:26:42.155138 8941 raft_consensus.cc:482] T 17ce77e57f7f4489809676f44fd21736 P 2165ea4f9946458d8204b0164cb2b5c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:26:42.155359 8941 raft_consensus.cc:504] T 17ce77e57f7f4489809676f44fd21736 P 2165ea4f9946458d8204b0164cb2b5c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a073fab4b2764065b0583d04312bfc4d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33149 } } peers { permanent_uuid: "c6738bbabd1d4f158acabb6eb177c156" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38973 } } peers { permanent_uuid: "2165ea4f9946458d8204b0164cb2b5c7" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36519 } }
I20260807 06:26:42.156646 8941 leader_election.cc:288] T 17ce77e57f7f4489809676f44fd21736 P 2165ea4f9946458d8204b0164cb2b5c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a073fab4b2764065b0583d04312bfc4d (127.15.254.3:33149), c6738bbabd1d4f158acabb6eb177c156 (127.15.254.2:38973)
I20260807 06:26:42.162323 8931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:42.156966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:43553 (local address 127.15.254.3:33149)
0807 06:26:42.157144 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:42.157149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:42.157386 (+ 237us) server_negotiation.cc:407] Connection header received
0807 06:26:42.157642 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:42.157645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:42.157696 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:42.157755 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:42.158247 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.159061 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.161068 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.161688 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.161798 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:42.161853 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:42.162123 (+ 270us) server_negotiation.cc:299] Negotiation successful
0807 06:26:42.162193 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 06:26:42.162323 8930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:42.157516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:38485 (local address 127.15.254.2:38973)
0807 06:26:42.157646 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:42.157650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:42.157795 (+ 145us) server_negotiation.cc:407] Connection header received
0807 06:26:42.157886 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:42.157888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:42.157929 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:42.158003 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:42.158253 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.159065 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.161039 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.161664 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.161798 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:42.161853 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:42.162121 (+ 268us) server_negotiation.cc:299] Negotiation successful
0807 06:26:42.162196 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 06:26:42.162366 8945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:42.157372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:38973 (local address 127.15.254.1:38485)
0807 06:26:42.157704 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:42.157717 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:42.157807 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:42.158021 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:42.158024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:42.158036 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:42.158138 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:42.158147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.159196 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.159201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:42.160879 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:42.160891 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.161803 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.161807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:42.161949 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:42.161982 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 06:26:42.162069 (+ 87us) client_negotiation.cc:240] Negotiation successful
0807 06:26:42.162168 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":113,"thread_start_us":106,"threads_started":1}
I20260807 06:26:42.162371 8944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:42.156888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:33149 (local address 127.15.254.1:43553)
0807 06:26:42.157284 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:42.157301 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:42.157470 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:42.157919 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:42.157921 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:42.157935 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:42.158101 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:42.158109 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.159196 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.159201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:42.160879 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:42.160891 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.161803 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.161807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:42.161997 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:42.162025 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 06:26:42.162094 (+ 69us) client_negotiation.cc:240] Negotiation successful
0807 06:26:42.162168 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":246,"thread_start_us":169,"threads_started":1}
I20260807 06:26:42.163231 8849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ce77e57f7f4489809676f44fd21736" candidate_uuid: "2165ea4f9946458d8204b0164cb2b5c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a073fab4b2764065b0583d04312bfc4d" is_pre_election: true
I20260807 06:26:42.163239 8657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ce77e57f7f4489809676f44fd21736" candidate_uuid: "2165ea4f9946458d8204b0164cb2b5c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6738bbabd1d4f158acabb6eb177c156" is_pre_election: true
I20260807 06:26:42.163544 8849 raft_consensus.cc:2454] T 17ce77e57f7f4489809676f44fd21736 P a073fab4b2764065b0583d04312bfc4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2165ea4f9946458d8204b0164cb2b5c7 in term 0.
I20260807 06:26:42.163573 8657 raft_consensus.cc:2454] T 17ce77e57f7f4489809676f44fd21736 P c6738bbabd1d4f158acabb6eb177c156 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2165ea4f9946458d8204b0164cb2b5c7 in term 0.
I20260807 06:26:42.164041 8376 leader_election.cc:302] T 17ce77e57f7f4489809676f44fd21736 P 2165ea4f9946458d8204b0164cb2b5c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2165ea4f9946458d8204b0164cb2b5c7, a073fab4b2764065b0583d04312bfc4d; no voters:
I20260807 06:26:42.164291 8941 raft_consensus.cc:2781] T 17ce77e57f7f4489809676f44fd21736 P 2165ea4f9946458d8204b0164cb2b5c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:26:42.164382 8941 raft_consensus.cc:482] T 17ce77e57f7f4489809676f44fd21736 P 2165ea4f9946458d8204b0164cb2b5c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:26:42.164428 8941 raft_consensus.cc:3037] T 17ce77e57f7f4489809676f44fd21736 P 2165ea4f9946458d8204b0164cb2b5c7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:42.165549 8941 raft_consensus.cc:504] T 17ce77e57f7f4489809676f44fd21736 P 2165ea4f9946458d8204b0164cb2b5c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a073fab4b2764065b0583d04312bfc4d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33149 } } peers { permanent_uuid: "c6738bbabd1d4f158acabb6eb177c156" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38973 } } peers { permanent_uuid: "2165ea4f9946458d8204b0164cb2b5c7" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36519 } }
I20260807 06:26:42.165886 8941 leader_election.cc:288] T 17ce77e57f7f4489809676f44fd21736 P 2165ea4f9946458d8204b0164cb2b5c7 [CANDIDATE]: Term 1 election: Requested vote from peers a073fab4b2764065b0583d04312bfc4d (127.15.254.3:33149), c6738bbabd1d4f158acabb6eb177c156 (127.15.254.2:38973)
I20260807 06:26:42.166290 8849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ce77e57f7f4489809676f44fd21736" candidate_uuid: "2165ea4f9946458d8204b0164cb2b5c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a073fab4b2764065b0583d04312bfc4d"
I20260807 06:26:42.166316 8657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ce77e57f7f4489809676f44fd21736" candidate_uuid: "2165ea4f9946458d8204b0164cb2b5c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6738bbabd1d4f158acabb6eb177c156"
I20260807 06:26:42.166429 8849 raft_consensus.cc:3037] T 17ce77e57f7f4489809676f44fd21736 P a073fab4b2764065b0583d04312bfc4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:42.166455 8657 raft_consensus.cc:3037] T 17ce77e57f7f4489809676f44fd21736 P c6738bbabd1d4f158acabb6eb177c156 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:42.167830 8657 raft_consensus.cc:2454] T 17ce77e57f7f4489809676f44fd21736 P c6738bbabd1d4f158acabb6eb177c156 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2165ea4f9946458d8204b0164cb2b5c7 in term 1.
I20260807 06:26:42.167867 8849 raft_consensus.cc:2454] T 17ce77e57f7f4489809676f44fd21736 P a073fab4b2764065b0583d04312bfc4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2165ea4f9946458d8204b0164cb2b5c7 in term 1.
I20260807 06:26:42.168138 8376 leader_election.cc:302] T 17ce77e57f7f4489809676f44fd21736 P 2165ea4f9946458d8204b0164cb2b5c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2165ea4f9946458d8204b0164cb2b5c7, c6738bbabd1d4f158acabb6eb177c156; no voters:
I20260807 06:26:42.168462 8941 raft_consensus.cc:2781] T 17ce77e57f7f4489809676f44fd21736 P 2165ea4f9946458d8204b0164cb2b5c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:26:42.168740 8941 raft_consensus.cc:686] T 17ce77e57f7f4489809676f44fd21736 P 2165ea4f9946458d8204b0164cb2b5c7 [term 1 LEADER]: Becoming Leader. State: Replica: 2165ea4f9946458d8204b0164cb2b5c7, State: Running, Role: LEADER
I20260807 06:26:42.169072 8941 consensus_queue.cc:237] T 17ce77e57f7f4489809676f44fd21736 P 2165ea4f9946458d8204b0164cb2b5c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a073fab4b2764065b0583d04312bfc4d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33149 } } peers { permanent_uuid: "c6738bbabd1d4f158acabb6eb177c156" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38973 } } peers { permanent_uuid: "2165ea4f9946458d8204b0164cb2b5c7" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36519 } }
I20260807 06:26:42.172137 8257 catalog_manager.cc:5495] T 17ce77e57f7f4489809676f44fd21736 P 2165ea4f9946458d8204b0164cb2b5c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2165ea4f9946458d8204b0164cb2b5c7 (127.15.254.1). New cstate: current_term: 1 leader_uuid: "2165ea4f9946458d8204b0164cb2b5c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a073fab4b2764065b0583d04312bfc4d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6738bbabd1d4f158acabb6eb177c156" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2165ea4f9946458d8204b0164cb2b5c7" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 36519 } health_report { overall_health: HEALTHY } } }
I20260807 06:26:42.191336 8932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:42.187679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37288 (local address 127.15.254.1:36519)
0807 06:26:42.187810 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:42.187817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:42.187834 (+ 17us) server_negotiation.cc:407] Connection header received
0807 06:26:42.187905 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:42.187907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:42.187965 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:42.188050 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:26:42.188285 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.188898 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.189767 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.190013 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.190097 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:42.190446 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:26:42.190859 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:26:42.190953 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:42.191159 (+ 206us) server_negotiation.cc:299] Negotiation successful
0807 06:26:42.191222 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 06:26:42.198118 8657 raft_consensus.cc:1261] T 17ce77e57f7f4489809676f44fd21736 P c6738bbabd1d4f158acabb6eb177c156 [term 1 FOLLOWER]: Refusing update from remote peer 2165ea4f9946458d8204b0164cb2b5c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:26:42.198118 8849 raft_consensus.cc:1261] T 17ce77e57f7f4489809676f44fd21736 P a073fab4b2764065b0583d04312bfc4d [term 1 FOLLOWER]: Refusing update from remote peer 2165ea4f9946458d8204b0164cb2b5c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:26:42.198994 8946 consensus_queue.cc:1035] T 17ce77e57f7f4489809676f44fd21736 P 2165ea4f9946458d8204b0164cb2b5c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a073fab4b2764065b0583d04312bfc4d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 33149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:26:42.199229 8941 consensus_queue.cc:1035] T 17ce77e57f7f4489809676f44fd21736 P 2165ea4f9946458d8204b0164cb2b5c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6738bbabd1d4f158acabb6eb177c156" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:26:42.216981 8257 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:26:42.217181 8257 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:26:42.219739 8257 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5b8e25e358541f58a62353a2ce9da86: Sending DeleteTablet for 3 replicas of tablet 17ce77e57f7f4489809676f44fd21736
I20260807 06:26:42.220640 8442 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ce77e57f7f4489809676f44fd21736 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:26:42 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:37278
I20260807 06:26:42.220758 8819 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ce77e57f7f4489809676f44fd21736 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:26:42 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:51832
I20260807 06:26:42.221297 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 8348
I20260807 06:26:42.221531 8629 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ce77e57f7f4489809676f44fd21736 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:26:42 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:50126
I20260807 06:26:42.223610 8970 tablet_replica.cc:330] T 17ce77e57f7f4489809676f44fd21736 P c6738bbabd1d4f158acabb6eb177c156: stopping tablet replica
I20260807 06:26:42.223959 8970 raft_consensus.cc:2229] T 17ce77e57f7f4489809676f44fd21736 P c6738bbabd1d4f158acabb6eb177c156 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:26:42.224270 8970 raft_consensus.cc:2258] T 17ce77e57f7f4489809676f44fd21736 P c6738bbabd1d4f158acabb6eb177c156 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:26:42.226228 8968 tablet_replica.cc:330] T 17ce77e57f7f4489809676f44fd21736 P a073fab4b2764065b0583d04312bfc4d: stopping tablet replica
I20260807 06:26:42.226698 8968 raft_consensus.cc:2229] T 17ce77e57f7f4489809676f44fd21736 P a073fab4b2764065b0583d04312bfc4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:26:42.227021 8970 ts_tablet_manager.cc:1905] T 17ce77e57f7f4489809676f44fd21736 P c6738bbabd1d4f158acabb6eb177c156: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:26:42.227121 8968 raft_consensus.cc:2258] T 17ce77e57f7f4489809676f44fd21736 P a073fab4b2764065b0583d04312bfc4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:26:42.229281 8968 ts_tablet_manager.cc:1905] T 17ce77e57f7f4489809676f44fd21736 P a073fab4b2764065b0583d04312bfc4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:26:42.230049 8970 ts_tablet_manager.cc:1918] T 17ce77e57f7f4489809676f44fd21736 P c6738bbabd1d4f158acabb6eb177c156: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:26:42.230181 8970 log.cc:1196] T 17ce77e57f7f4489809676f44fd21736 P c6738bbabd1d4f158acabb6eb177c156: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/17ce77e57f7f4489809676f44fd21736
I20260807 06:26:42.230510 8970 ts_tablet_manager.cc:1939] T 17ce77e57f7f4489809676f44fd21736 P c6738bbabd1d4f158acabb6eb177c156: Deleting consensus metadata
I20260807 06:26:42.231808 8239 catalog_manager.cc:4841] TS c6738bbabd1d4f158acabb6eb177c156 (127.15.254.2:38973): tablet 17ce77e57f7f4489809676f44fd21736 (table test-table [id=5e1353fde38b44078d494200f2b5952d]) successfully deleted
I20260807 06:26:42.233209 8968 ts_tablet_manager.cc:1918] T 17ce77e57f7f4489809676f44fd21736 P a073fab4b2764065b0583d04312bfc4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:26:42.233319 8968 log.cc:1196] T 17ce77e57f7f4489809676f44fd21736 P a073fab4b2764065b0583d04312bfc4d: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/17ce77e57f7f4489809676f44fd21736
I20260807 06:26:42.233678 8968 ts_tablet_manager.cc:1939] T 17ce77e57f7f4489809676f44fd21736 P a073fab4b2764065b0583d04312bfc4d: Deleting consensus metadata
I20260807 06:26:42.235005 8239 catalog_manager.cc:4841] TS a073fab4b2764065b0583d04312bfc4d (127.15.254.3:33149): tablet 17ce77e57f7f4489809676f44fd21736 (table test-table [id=5e1353fde38b44078d494200f2b5952d]) successfully deleted
W20260807 06:26:42.242125 8237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.254.1:36519 (error 108)
I20260807 06:26:42.242403 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 8537
I20260807 06:26:42.251319 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:42.250877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:36519 (local address 127.0.0.1:37294)
0807 06:26:42.251116 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:42.251208 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.1:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260807 06:26:42.251549 8237 catalog_manager.cc:4584] TS 2165ea4f9946458d8204b0164cb2b5c7 (127.15.254.1:36519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 17ce77e57f7f4489809676f44fd21736: Network error: Client connection negotiation failed: client connection to 127.15.254.1:36519: connect: Connection refused (error 111)
I20260807 06:26:42.257158 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 8729
I20260807 06:26:42.274200 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 8214
2026-08-07T06:26:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2625 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-07T06:26:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:26:42Z Disabled control of system clock
I20260807 06:26:42.316447 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:38225
--webserver_interface=127.15.254.62
--webserver_port=0
--builtin_ntp_servers=127.15.254.20:34939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:38225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 06:26:42.433184 8977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:42.433476 8977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:42.433516 8977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:42.437633 8977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:26:42.437705 8977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:42.437721 8977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:42.437737 8977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:26:42.437753 8977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:26:42.442728 8977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:34939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:38225
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:38225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:42.443879 8977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:42.445408 8977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:42.454845 8984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:42.454845 8985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:42.454840 8987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:42.455031 8977 server_base.cc:1034] running on GCE node
I20260807 06:26:42.455694 8977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:42.456885 8977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:42.458107 8977 hybrid_clock.cc:648] HybridClock initialized: now 1786084002458072 us; error 49 us; skew 500 ppm
I20260807 06:26:42.460120 8977 webserver.cc:460] Webserver started at http://127.15.254.62:35915/ using document root <none> and password file <none>
I20260807 06:26:42.460795 8977 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:42.460891 8977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:42.461143 8977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:42.463296 8977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "a2d3ad9cad0e4df9a68887b9838f10c6"
format_stamp: "Formatted at 2026-08-07 06:26:42 on dist-test-slave-44d2"
server_key: "6df664a5a2b19881d9d6d24d1f576f15"
server_key_iv: "618f29502d9885f7215ebcc24d8728b8"
server_key_version: "encryptionkey@0"
I20260807 06:26:42.463878 8977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "a2d3ad9cad0e4df9a68887b9838f10c6"
format_stamp: "Formatted at 2026-08-07 06:26:42 on dist-test-slave-44d2"
server_key: "6df664a5a2b19881d9d6d24d1f576f15"
server_key_iv: "618f29502d9885f7215ebcc24d8728b8"
server_key_version: "encryptionkey@0"
I20260807 06:26:42.467752 8977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 06:26:42.470618 8994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:42.472085 8977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:26:42.472256 8977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "a2d3ad9cad0e4df9a68887b9838f10c6"
format_stamp: "Formatted at 2026-08-07 06:26:42 on dist-test-slave-44d2"
server_key: "6df664a5a2b19881d9d6d24d1f576f15"
server_key_iv: "618f29502d9885f7215ebcc24d8728b8"
server_key_version: "encryptionkey@0"
I20260807 06:26:42.472385 8977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:42.489508 8977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:42.490257 8977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:42.490471 8977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:42.498338 8977 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:38225
I20260807 06:26:42.498410 9068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:38225 every 8 connection(s)
I20260807 06:26:42.499389 8977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:26:42.502795 9069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:42.503402 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 8977
I20260807 06:26:42.503556 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:26:42.503890 16376 external_mini_cluster.cc:1428] Setting key 47dc4e8f889bb2abf3fcf867357d453f
I20260807 06:26:42.509495 9069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d3ad9cad0e4df9a68887b9838f10c6: Bootstrap starting.
I20260807 06:26:42.512264 9069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2d3ad9cad0e4df9a68887b9838f10c6: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:42.513207 9069 log.cc:824] T 00000000000000000000000000000000 P a2d3ad9cad0e4df9a68887b9838f10c6: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:42.513688 9074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:42.505404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44640 (local address 127.15.254.62:38225)
0807 06:26:42.506042 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:42.506056 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:42.506092 (+ 36us) server_negotiation.cc:407] Connection header received
0807 06:26:42.506754 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:42.506771 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:42.507166 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:42.507451 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:42.507862 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.508931 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.510125 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.510588 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.510705 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:42.511656 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:42.511661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:42.511673 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:26:42.512061 (+ 388us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:42.512115 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:26:42.512134 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:42.512212 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:42.512518 (+ 306us) server_negotiation.cc:299] Negotiation successful
0807 06:26:42.512789 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":127,"threads_started":1}
I20260807 06:26:42.515594 9069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d3ad9cad0e4df9a68887b9838f10c6: No bootstrap required, opened a new log
I20260807 06:26:42.520183 9069 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2d3ad9cad0e4df9a68887b9838f10c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d3ad9cad0e4df9a68887b9838f10c6" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 38225 } }
I20260807 06:26:42.520458 9069 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2d3ad9cad0e4df9a68887b9838f10c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:42.520576 9069 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2d3ad9cad0e4df9a68887b9838f10c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2d3ad9cad0e4df9a68887b9838f10c6, State: Initialized, Role: FOLLOWER
I20260807 06:26:42.521013 9069 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2d3ad9cad0e4df9a68887b9838f10c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d3ad9cad0e4df9a68887b9838f10c6" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 38225 } }
I20260807 06:26:42.521137 9069 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2d3ad9cad0e4df9a68887b9838f10c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:26:42.521180 9069 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2d3ad9cad0e4df9a68887b9838f10c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:26:42.521247 9069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2d3ad9cad0e4df9a68887b9838f10c6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:42.522193 9069 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2d3ad9cad0e4df9a68887b9838f10c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d3ad9cad0e4df9a68887b9838f10c6" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 38225 } }
I20260807 06:26:42.522504 9069 leader_election.cc:302] T 00000000000000000000000000000000 P a2d3ad9cad0e4df9a68887b9838f10c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2d3ad9cad0e4df9a68887b9838f10c6; no voters:
I20260807 06:26:42.522779 9069 leader_election.cc:288] T 00000000000000000000000000000000 P a2d3ad9cad0e4df9a68887b9838f10c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:26:42.522936 9077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2d3ad9cad0e4df9a68887b9838f10c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:26:42.523229 9077 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2d3ad9cad0e4df9a68887b9838f10c6 [term 1 LEADER]: Becoming Leader. State: Replica: a2d3ad9cad0e4df9a68887b9838f10c6, State: Running, Role: LEADER
I20260807 06:26:42.523564 9077 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2d3ad9cad0e4df9a68887b9838f10c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d3ad9cad0e4df9a68887b9838f10c6" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 38225 } }
I20260807 06:26:42.524153 9069 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2d3ad9cad0e4df9a68887b9838f10c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:26:42.525470 9078 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d3ad9cad0e4df9a68887b9838f10c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2d3ad9cad0e4df9a68887b9838f10c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d3ad9cad0e4df9a68887b9838f10c6" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 38225 } } }
I20260807 06:26:42.525609 9078 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d3ad9cad0e4df9a68887b9838f10c6 [sys.catalog]: This master's current role is: LEADER
I20260807 06:26:42.525911 9079 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d3ad9cad0e4df9a68887b9838f10c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2d3ad9cad0e4df9a68887b9838f10c6. Latest consensus state: current_term: 1 leader_uuid: "a2d3ad9cad0e4df9a68887b9838f10c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d3ad9cad0e4df9a68887b9838f10c6" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 38225 } } }
I20260807 06:26:42.526196 9079 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d3ad9cad0e4df9a68887b9838f10c6 [sys.catalog]: This master's current role is: LEADER
W20260807 06:26:42.529584 9099 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2d3ad9cad0e4df9a68887b9838f10c6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:26:42.529660 9099 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:26:42.529744 9091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:26:42.530463 9091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:26:42.535826 9091 catalog_manager.cc:1347] Generated new cluster ID: 49baf7755f6a484f99b6457ee96799ff
I20260807 06:26:42.535912 9091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:26:42.557739 9091 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:26:42.559500 9091 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:26:42.577557 9091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2d3ad9cad0e4df9a68887b9838f10c6: Generated new TSK 0
I20260807 06:26:42.578617 9091 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:26:42.593014 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:0
--local_ip_for_outbound_sockets=127.15.254.1
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:38225
--builtin_ntp_servers=127.15.254.20:34939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 06:26:42.715097 9105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:42.715384 9105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:42.715425 9105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:42.719563 9105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:42.719632 9105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:42.719722 9105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:26:42.724761 9105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:34939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:38225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:42.726043 9105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:42.727557 9105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:42.737615 9110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:42.737615 9117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:42.737799 9105 server_base.cc:1034] running on GCE node
W20260807 06:26:42.738071 9113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:42.738502 9105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:42.739166 9105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:42.740367 9105 hybrid_clock.cc:648] HybridClock initialized: now 1786084002740350 us; error 26 us; skew 500 ppm
I20260807 06:26:42.742473 9105 webserver.cc:460] Webserver started at http://127.15.254.1:35023/ using document root <none> and password file <none>
I20260807 06:26:42.743196 9105 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:42.743305 9105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:42.743590 9105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:42.745965 9105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-0/data/instance:
uuid: "02733a436b1e44caa380acca6f848a86"
format_stamp: "Formatted at 2026-08-07 06:26:42 on dist-test-slave-44d2"
server_key: "dbde12dbb733693ddfcb771d45e02c5d"
server_key_iv: "ed021c739b24434a024034b9b77e7eab"
server_key_version: "encryptionkey@0"
I20260807 06:26:42.746603 9105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance:
uuid: "02733a436b1e44caa380acca6f848a86"
format_stamp: "Formatted at 2026-08-07 06:26:42 on dist-test-slave-44d2"
server_key: "dbde12dbb733693ddfcb771d45e02c5d"
server_key_iv: "ed021c739b24434a024034b9b77e7eab"
server_key_version: "encryptionkey@0"
I20260807 06:26:42.750705 9105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:26:42.753576 9125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:42.754963 9105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:26:42.755132 9105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "02733a436b1e44caa380acca6f848a86"
format_stamp: "Formatted at 2026-08-07 06:26:42 on dist-test-slave-44d2"
server_key: "dbde12dbb733693ddfcb771d45e02c5d"
server_key_iv: "ed021c739b24434a024034b9b77e7eab"
server_key_version: "encryptionkey@0"
I20260807 06:26:42.755261 9105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:42.778488 9105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:42.779253 9105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:42.779471 9105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:42.780118 9105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:26:42.781206 9105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:26:42.781291 9105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:42.781384 9105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:26:42.781442 9105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:42.791044 9105 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:42073
I20260807 06:26:42.791118 9284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:42073 every 8 connection(s)
I20260807 06:26:42.791994 9105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:26:42.801749 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 9105
I20260807 06:26:42.801913 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance
I20260807 06:26:42.802217 16376 external_mini_cluster.cc:1428] Setting key f1f438f19d194317f5e15d376fca0677
I20260807 06:26:42.802431 9074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:42.794154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:48449 (local address 127.15.254.62:38225)
0807 06:26:42.794327 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:42.794332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:42.795701 (+ 1369us) server_negotiation.cc:407] Connection header received
0807 06:26:42.797118 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:42.797121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:42.797174 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:42.797270 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:42.798241 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.798861 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.800137 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.800381 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.800576 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:42.801760 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:42.801763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:42.801765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:26:42.801778 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:42.801797 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:26:42.801805 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:42.802016 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:42.802204 (+ 188us) server_negotiation.cc:299] Negotiation successful
0807 06:26:42.802275 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 06:26:42.803476 9288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:42.794668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:38225 (local address 127.15.254.1:48449)
0807 06:26:42.795423 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:42.795467 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:42.796878 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:42.797436 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:42.797442 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:42.797530 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:26:42.798063 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:42.798078 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.799090 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.799092 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:42.799995 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:42.800004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:42.800561 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:42.800564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:42.800706 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:42.801415 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:26:42.801417 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:26:42.801476 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:26:42.801567 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:26:42.802016 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:26:42.802030 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:26:42.802039 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 06:26:42.802188 (+ 149us) client_negotiation.cc:240] Negotiation successful
0807 06:26:42.802511 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":136,"threads_started":1}
I20260807 06:26:42.804593 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:0
--local_ip_for_outbound_sockets=127.15.254.2
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:38225
--builtin_ntp_servers=127.15.254.20:34939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 06:26:42.804899 9285 heartbeater.cc:347] Connected to a master server at 127.15.254.62:38225
I20260807 06:26:42.805153 9285 heartbeater.cc:464] Registering TS with master...
I20260807 06:26:42.805694 9285 heartbeater.cc:511] Master 127.15.254.62:38225 requested a full tablet report, sending...
I20260807 06:26:42.807329 9021 ts_manager.cc:194] Registered new tserver with Master: 02733a436b1e44caa380acca6f848a86 (127.15.254.1:42073)
I20260807 06:26:42.809000 9021 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.254.1:48449
W20260807 06:26:42.924207 9290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:42.924548 9290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:42.924611 9290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:42.928999 9290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:42.929086 9290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:42.929147 9290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:26:42.934655 9290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:34939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:38225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:42.935738 9290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:42.937196 9290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:42.947839 9304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:42.947868 9299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:42.948091 9290 server_base.cc:1034] running on GCE node
W20260807 06:26:42.947921 9301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:42.948635 9290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:42.949333 9290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:42.950519 9290 hybrid_clock.cc:648] HybridClock initialized: now 1786084002950502 us; error 26 us; skew 500 ppm
I20260807 06:26:42.952704 9290 webserver.cc:460] Webserver started at http://127.15.254.2:38049/ using document root <none> and password file <none>
I20260807 06:26:42.953368 9290 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:42.953439 9290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:42.953662 9290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:42.955786 9290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-1/data/instance:
uuid: "bce5cbe7e1be49eaabca0fb8bd284728"
format_stamp: "Formatted at 2026-08-07 06:26:42 on dist-test-slave-44d2"
server_key: "9e28b315e33b73995d7de39bc2b23a2a"
server_key_iv: "22560f315744bfffa111423fce16b70f"
server_key_version: "encryptionkey@0"
I20260807 06:26:42.956386 9290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance:
uuid: "bce5cbe7e1be49eaabca0fb8bd284728"
format_stamp: "Formatted at 2026-08-07 06:26:42 on dist-test-slave-44d2"
server_key: "9e28b315e33b73995d7de39bc2b23a2a"
server_key_iv: "22560f315744bfffa111423fce16b70f"
server_key_version: "encryptionkey@0"
I20260807 06:26:42.960459 9290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260807 06:26:42.963210 9312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:42.964720 9290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 06:26:42.964870 9290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "bce5cbe7e1be49eaabca0fb8bd284728"
format_stamp: "Formatted at 2026-08-07 06:26:42 on dist-test-slave-44d2"
server_key: "9e28b315e33b73995d7de39bc2b23a2a"
server_key_iv: "22560f315744bfffa111423fce16b70f"
server_key_version: "encryptionkey@0"
I20260807 06:26:42.965004 9290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:42.981563 9290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:42.982339 9290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:42.982571 9290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:42.983251 9290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:26:42.984287 9290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:26:42.984362 9290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:42.984450 9290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:26:42.984524 9290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260807 06:26:42.994653 9290 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:38335
I20260807 06:26:42.994705 9461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:38335 every 8 connection(s)
I20260807 06:26:42.995664 9290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:26:43.003537 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 9290
I20260807 06:26:43.003693 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance
I20260807 06:26:43.004065 16376 external_mini_cluster.cc:1428] Setting key b402993fc91159b37757c9b1e8981000
I20260807 06:26:43.006246 9074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:42.997792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:38975 (local address 127.15.254.62:38225)
0807 06:26:42.998007 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:42.998011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:42.999193 (+ 1182us) server_negotiation.cc:407] Connection header received
0807 06:26:43.000761 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:43.000765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:43.000829 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:43.000940 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:43.001943 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.002494 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.004038 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.004345 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.004465 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:43.005758 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:43.005762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:43.005765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:26:43.005778 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:43.005812 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:26:43.005822 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:43.005879 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:43.006077 (+ 198us) server_negotiation.cc:299] Negotiation successful
0807 06:26:43.006133 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 06:26:43.006541 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:0
--local_ip_for_outbound_sockets=127.15.254.3
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:38225
--builtin_ntp_servers=127.15.254.20:34939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 06:26:43.007642 9465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:42.998304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:38225 (local address 127.15.254.2:38975)
0807 06:26:42.999028 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:42.999072 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:43.000525 (+ 1453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:43.001137 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:43.001144 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:43.001234 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:26:43.001779 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:43.001793 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.002649 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.002652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:43.003874 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:43.003889 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.004574 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.004579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:43.004740 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:43.005442 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:26:43.005445 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:26:43.005539 (+ 94us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:26:43.005614 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:26:43.005907 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:26:43.005918 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:26:43.005924 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 06:26:43.006071 (+ 147us) client_negotiation.cc:240] Negotiation successful
0807 06:26:43.006412 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":109,"threads_started":1}
I20260807 06:26:43.009195 9462 heartbeater.cc:347] Connected to a master server at 127.15.254.62:38225
I20260807 06:26:43.009516 9462 heartbeater.cc:464] Registering TS with master...
I20260807 06:26:43.010255 9462 heartbeater.cc:511] Master 127.15.254.62:38225 requested a full tablet report, sending...
I20260807 06:26:43.011628 9021 ts_manager.cc:194] Registered new tserver with Master: bce5cbe7e1be49eaabca0fb8bd284728 (127.15.254.2:38335)
I20260807 06:26:43.012316 9021 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.254.2:38975
W20260807 06:26:43.130389 9467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:43.130702 9467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:43.130754 9467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:43.135254 9467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:43.135317 9467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:43.135375 9467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:26:43.140372 9467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:34939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:38225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:43.141575 9467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:43.143108 9467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:43.153491 9477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:43.153570 9467 server_base.cc:1034] running on GCE node
W20260807 06:26:43.153548 9476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:43.153745 9479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:43.154309 9467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:43.155079 9467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:43.156279 9467 hybrid_clock.cc:648] HybridClock initialized: now 1786084003156217 us; error 72 us; skew 500 ppm
I20260807 06:26:43.158565 9467 webserver.cc:460] Webserver started at http://127.15.254.3:37065/ using document root <none> and password file <none>
I20260807 06:26:43.159222 9467 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:43.159296 9467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:43.159566 9467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:43.161875 9467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-2/data/instance:
uuid: "e51a727d774f4e9193013fa6b010d10d"
format_stamp: "Formatted at 2026-08-07 06:26:43 on dist-test-slave-44d2"
server_key: "2e8de40a4f25110b5e945b57b7158e4e"
server_key_iv: "0086eb0370a3fbc48bb9f43984a868a8"
server_key_version: "encryptionkey@0"
I20260807 06:26:43.162504 9467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance:
uuid: "e51a727d774f4e9193013fa6b010d10d"
format_stamp: "Formatted at 2026-08-07 06:26:43 on dist-test-slave-44d2"
server_key: "2e8de40a4f25110b5e945b57b7158e4e"
server_key_iv: "0086eb0370a3fbc48bb9f43984a868a8"
server_key_version: "encryptionkey@0"
I20260807 06:26:43.166637 9467 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260807 06:26:43.169813 9487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:43.171285 9467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:26:43.171466 9467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "e51a727d774f4e9193013fa6b010d10d"
format_stamp: "Formatted at 2026-08-07 06:26:43 on dist-test-slave-44d2"
server_key: "2e8de40a4f25110b5e945b57b7158e4e"
server_key_iv: "0086eb0370a3fbc48bb9f43984a868a8"
server_key_version: "encryptionkey@0"
I20260807 06:26:43.171592 9467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:43.196368 9467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:43.197319 9467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:43.197566 9467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:43.198277 9467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:26:43.199375 9467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:26:43.199432 9467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:43.199481 9467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:26:43.199548 9467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:43.210383 9467 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:36209
I20260807 06:26:43.210408 9640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:36209 every 8 connection(s)
I20260807 06:26:43.211472 9467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:26:43.216328 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 9467
I20260807 06:26:43.216522 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance
I20260807 06:26:43.216912 16376 external_mini_cluster.cc:1428] Setting key 04a7ce20650f3b2174be717d9d3fa464
I20260807 06:26:43.222150 9074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:43.213585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:39165 (local address 127.15.254.62:38225)
0807 06:26:43.213778 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:43.213783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:43.214995 (+ 1212us) server_negotiation.cc:407] Connection header received
0807 06:26:43.216667 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:43.216670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:43.216722 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:43.216805 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:43.217953 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.218621 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.220003 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.220237 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.220362 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:43.221619 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:43.221623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:43.221626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:26:43.221640 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:43.221668 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:26:43.221676 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:43.221754 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:43.221931 (+ 177us) server_negotiation.cc:299] Negotiation successful
0807 06:26:43.221999 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 06:26:43.223261 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:43.214046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:38225 (local address 127.15.254.3:39165)
0807 06:26:43.214763 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:43.214808 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:43.216375 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:43.217040 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:43.217051 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:43.217161 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:26:43.217782 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:43.217804 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.218803 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.218806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:43.219873 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:43.219883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.220402 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.220407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:43.220646 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:43.221338 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:26:43.221340 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:26:43.221385 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:26:43.221455 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:26:43.221769 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:26:43.221782 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:26:43.221792 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 06:26:43.221931 (+ 139us) client_negotiation.cc:240] Negotiation successful
0807 06:26:43.222240 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":121,"threads_started":1}
I20260807 06:26:43.224620 9642 heartbeater.cc:347] Connected to a master server at 127.15.254.62:38225
I20260807 06:26:43.225028 9642 heartbeater.cc:464] Registering TS with master...
I20260807 06:26:43.225710 9642 heartbeater.cc:511] Master 127.15.254.62:38225 requested a full tablet report, sending...
I20260807 06:26:43.226868 9021 ts_manager.cc:194] Registered new tserver with Master: e51a727d774f4e9193013fa6b010d10d (127.15.254.3:36209)
I20260807 06:26:43.227545 9021 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.254.3:39165
I20260807 06:26:43.231555 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:26:43.237744 9074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:43.233859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44642 (local address 127.15.254.62:38225)
0807 06:26:43.234017 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:43.234022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:43.234249 (+ 227us) server_negotiation.cc:407] Connection header received
0807 06:26:43.234349 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:43.234351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:43.234394 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:43.234467 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:43.234740 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.235295 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.236100 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.236306 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.236384 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:43.237355 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:43.237359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:43.237361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:26:43.237374 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:43.237398 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:26:43.237408 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:43.237482 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:43.237574 (+ 92us) server_negotiation.cc:299] Negotiation successful
0807 06:26:43.237630 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 06:26:43.241048 9021 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:26:43.244305 9021 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:26:43.262355 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:43.254329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:38335 (local address 127.0.0.1:56062)
0807 06:26:43.255088 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:43.255209 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:43.255362 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:43.256782 (+ 1420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:43.256786 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:43.256807 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:43.257005 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:43.257014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.258747 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.258750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:43.260391 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:43.260402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.261930 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.261934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:43.262031 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:43.262052 (+ 21us) client_negotiation.cc:769] Sending connection context
0807 06:26:43.262122 (+ 70us) client_negotiation.cc:240] Negotiation successful
0807 06:26:43.262179 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":294,"thread_start_us":87,"threads_started":1}
I20260807 06:26:43.262413 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:43.255251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33834 (local address 127.15.254.1:42073)
0807 06:26:43.255860 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:43.255868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:43.256226 (+ 358us) server_negotiation.cc:407] Connection header received
0807 06:26:43.256327 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:43.256333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:43.256604 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:43.256781 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:43.257170 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.258399 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.260681 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.261459 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.261624 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:43.261859 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:43.262068 (+ 209us) server_negotiation.cc:299] Negotiation successful
0807 06:26:43.262182 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":118,"threads_started":1}
I20260807 06:26:43.262353 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:43.254515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:36209 (local address 127.0.0.1:47144)
0807 06:26:43.255306 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:43.255323 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:43.255456 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:43.256512 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:43.256523 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:43.256557 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:43.256716 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:43.256726 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.258863 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.258866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:43.260465 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:43.260504 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.261661 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.261665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:43.261790 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:43.261872 (+ 82us) client_negotiation.cc:769] Sending connection context
0807 06:26:43.261975 (+ 103us) client_negotiation.cc:240] Negotiation successful
0807 06:26:43.262124 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":83,"thread_start_us":130,"threads_started":1}
I20260807 06:26:43.263151 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:43.254764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:42073 (local address 127.0.0.1:33834)
0807 06:26:43.255998 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:43.256016 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:43.256116 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:43.256768 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:43.256771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:43.256792 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:43.257013 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:43.257020 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.259020 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.259023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:43.260573 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:43.260584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.261629 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.261633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:43.261823 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:43.261874 (+ 51us) client_negotiation.cc:769] Sending connection context
0807 06:26:43.262030 (+ 156us) client_negotiation.cc:240] Negotiation successful
0807 06:26:43.263011 (+ 981us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1152,"mutex_wait_us":190,"spinlock_wait_cycles":29952,"thread_start_us":114,"threads_started":1}
I20260807 06:26:43.263351 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:43.254521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56062 (local address 127.15.254.2:38335)
0807 06:26:43.256116 (+ 1595us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:43.256128 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:43.256152 (+ 24us) server_negotiation.cc:407] Connection header received
0807 06:26:43.256234 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:43.256240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:43.256434 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:43.256618 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:43.257184 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.258476 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.260558 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.261498 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.261631 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:43.261860 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:43.263034 (+ 1174us) server_negotiation.cc:299] Negotiation successful
0807 06:26:43.263153 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1482,"thread_start_us":136,"threads_started":1}
I20260807 06:26:43.263458 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:43.255279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47144 (local address 127.15.254.3:36209)
0807 06:26:43.255833 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:43.255848 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:43.255881 (+ 33us) server_negotiation.cc:407] Connection header received
0807 06:26:43.256000 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:43.256006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:43.256261 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:43.256893 (+ 632us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:43.257055 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.258530 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.260612 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.261496 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.262481 (+ 985us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:43.263111 (+ 630us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:43.263164 (+ 53us) server_negotiation.cc:299] Negotiation successful
0807 06:26:43.263266 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":195,"threads_started":1}
I20260807 06:26:43.266520 9192 tablet_service.cc:1467] Processing CreateTablet for tablet 727156c4134647f394cd45f70824b436 (DEFAULT_TABLE table=test-table [id=856cc30942094f6494560eb83ecf7528]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:26:43.266551 9545 tablet_service.cc:1467] Processing CreateTablet for tablet 727156c4134647f394cd45f70824b436 (DEFAULT_TABLE table=test-table [id=856cc30942094f6494560eb83ecf7528]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:26:43.266551 9369 tablet_service.cc:1467] Processing CreateTablet for tablet 727156c4134647f394cd45f70824b436 (DEFAULT_TABLE table=test-table [id=856cc30942094f6494560eb83ecf7528]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:26:43.267741 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 727156c4134647f394cd45f70824b436. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:43.267774 9369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 727156c4134647f394cd45f70824b436. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:43.267774 9545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 727156c4134647f394cd45f70824b436. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:43.275532 9664 tablet_bootstrap.cc:492] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86: Bootstrap starting.
I20260807 06:26:43.277153 9665 tablet_bootstrap.cc:492] T 727156c4134647f394cd45f70824b436 P e51a727d774f4e9193013fa6b010d10d: Bootstrap starting.
I20260807 06:26:43.279109 9664 tablet_bootstrap.cc:654] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:43.279747 9665 tablet_bootstrap.cc:654] T 727156c4134647f394cd45f70824b436 P e51a727d774f4e9193013fa6b010d10d: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:43.280193 9664 log.cc:824] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:43.280766 9665 log.cc:824] T 727156c4134647f394cd45f70824b436 P e51a727d774f4e9193013fa6b010d10d: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:43.281031 9666 tablet_bootstrap.cc:492] T 727156c4134647f394cd45f70824b436 P bce5cbe7e1be49eaabca0fb8bd284728: Bootstrap starting.
I20260807 06:26:43.282626 9664 tablet_bootstrap.cc:492] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86: No bootstrap required, opened a new log
I20260807 06:26:43.282902 9664 ts_tablet_manager.cc:1397] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260807 06:26:43.283102 9665 tablet_bootstrap.cc:492] T 727156c4134647f394cd45f70824b436 P e51a727d774f4e9193013fa6b010d10d: No bootstrap required, opened a new log
I20260807 06:26:43.283317 9665 ts_tablet_manager.cc:1397] T 727156c4134647f394cd45f70824b436 P e51a727d774f4e9193013fa6b010d10d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 06:26:43.283514 9666 tablet_bootstrap.cc:654] T 727156c4134647f394cd45f70824b436 P bce5cbe7e1be49eaabca0fb8bd284728: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:43.284607 9666 log.cc:824] T 727156c4134647f394cd45f70824b436 P bce5cbe7e1be49eaabca0fb8bd284728: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:43.287202 9664 raft_consensus.cc:348] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce5cbe7e1be49eaabca0fb8bd284728" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38335 } } peers { permanent_uuid: "e51a727d774f4e9193013fa6b010d10d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36209 } } peers { permanent_uuid: "02733a436b1e44caa380acca6f848a86" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42073 } }
I20260807 06:26:43.287330 9665 raft_consensus.cc:348] T 727156c4134647f394cd45f70824b436 P e51a727d774f4e9193013fa6b010d10d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce5cbe7e1be49eaabca0fb8bd284728" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38335 } } peers { permanent_uuid: "e51a727d774f4e9193013fa6b010d10d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36209 } } peers { permanent_uuid: "02733a436b1e44caa380acca6f848a86" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42073 } }
I20260807 06:26:43.287513 9664 raft_consensus.cc:374] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:43.287591 9664 raft_consensus.cc:729] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02733a436b1e44caa380acca6f848a86, State: Initialized, Role: FOLLOWER
I20260807 06:26:43.287603 9665 raft_consensus.cc:374] T 727156c4134647f394cd45f70824b436 P e51a727d774f4e9193013fa6b010d10d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:43.287668 9665 raft_consensus.cc:729] T 727156c4134647f394cd45f70824b436 P e51a727d774f4e9193013fa6b010d10d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e51a727d774f4e9193013fa6b010d10d, State: Initialized, Role: FOLLOWER
I20260807 06:26:43.288183 9664 consensus_queue.cc:260] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce5cbe7e1be49eaabca0fb8bd284728" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38335 } } peers { permanent_uuid: "e51a727d774f4e9193013fa6b010d10d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36209 } } peers { permanent_uuid: "02733a436b1e44caa380acca6f848a86" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42073 } }
I20260807 06:26:43.288182 9665 consensus_queue.cc:260] T 727156c4134647f394cd45f70824b436 P e51a727d774f4e9193013fa6b010d10d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce5cbe7e1be49eaabca0fb8bd284728" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38335 } } peers { permanent_uuid: "e51a727d774f4e9193013fa6b010d10d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36209 } } peers { permanent_uuid: "02733a436b1e44caa380acca6f848a86" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42073 } }
I20260807 06:26:43.289049 9642 heartbeater.cc:503] Master 127.15.254.62:38225 was elected leader, sending a full tablet report...
I20260807 06:26:43.289384 9665 ts_tablet_manager.cc:1428] T 727156c4134647f394cd45f70824b436 P e51a727d774f4e9193013fa6b010d10d: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260807 06:26:43.289386 9285 heartbeater.cc:503] Master 127.15.254.62:38225 was elected leader, sending a full tablet report...
I20260807 06:26:43.289402 9664 ts_tablet_manager.cc:1428] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260807 06:26:43.291224 9666 tablet_bootstrap.cc:492] T 727156c4134647f394cd45f70824b436 P bce5cbe7e1be49eaabca0fb8bd284728: No bootstrap required, opened a new log
I20260807 06:26:43.291465 9666 ts_tablet_manager.cc:1397] T 727156c4134647f394cd45f70824b436 P bce5cbe7e1be49eaabca0fb8bd284728: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
W20260807 06:26:43.293324 9287 tablet.cc:2369] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:26:43.296311 9666 raft_consensus.cc:348] T 727156c4134647f394cd45f70824b436 P bce5cbe7e1be49eaabca0fb8bd284728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce5cbe7e1be49eaabca0fb8bd284728" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38335 } } peers { permanent_uuid: "e51a727d774f4e9193013fa6b010d10d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36209 } } peers { permanent_uuid: "02733a436b1e44caa380acca6f848a86" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42073 } }
I20260807 06:26:43.296659 9666 raft_consensus.cc:374] T 727156c4134647f394cd45f70824b436 P bce5cbe7e1be49eaabca0fb8bd284728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:43.296729 9666 raft_consensus.cc:729] T 727156c4134647f394cd45f70824b436 P bce5cbe7e1be49eaabca0fb8bd284728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce5cbe7e1be49eaabca0fb8bd284728, State: Initialized, Role: FOLLOWER
I20260807 06:26:43.297199 9666 consensus_queue.cc:260] T 727156c4134647f394cd45f70824b436 P bce5cbe7e1be49eaabca0fb8bd284728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce5cbe7e1be49eaabca0fb8bd284728" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38335 } } peers { permanent_uuid: "e51a727d774f4e9193013fa6b010d10d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36209 } } peers { permanent_uuid: "02733a436b1e44caa380acca6f848a86" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42073 } }
I20260807 06:26:43.297986 9462 heartbeater.cc:503] Master 127.15.254.62:38225 was elected leader, sending a full tablet report...
I20260807 06:26:43.298230 9666 ts_tablet_manager.cc:1428] T 727156c4134647f394cd45f70824b436 P bce5cbe7e1be49eaabca0fb8bd284728: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260807 06:26:43.462553 9644 tablet.cc:2369] T 727156c4134647f394cd45f70824b436 P e51a727d774f4e9193013fa6b010d10d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 06:26:43.497169 9463 tablet.cc:2369] T 727156c4134647f394cd45f70824b436 P bce5cbe7e1be49eaabca0fb8bd284728: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:26:43.524204 9671 raft_consensus.cc:482] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:26:43.524591 9671 raft_consensus.cc:504] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce5cbe7e1be49eaabca0fb8bd284728" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38335 } } peers { permanent_uuid: "e51a727d774f4e9193013fa6b010d10d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36209 } } peers { permanent_uuid: "02733a436b1e44caa380acca6f848a86" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42073 } }
I20260807 06:26:43.526403 9671 leader_election.cc:288] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bce5cbe7e1be49eaabca0fb8bd284728 (127.15.254.2:38335), e51a727d774f4e9193013fa6b010d10d (127.15.254.3:36209)
I20260807 06:26:43.532637 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:43.526915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:49525 (local address 127.15.254.3:36209)
0807 06:26:43.527230 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:43.527236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:43.527644 (+ 408us) server_negotiation.cc:407] Connection header received
0807 06:26:43.527915 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:43.527918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:43.527981 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:43.528064 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:43.528536 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.529259 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.531069 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.531700 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.531804 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:43.531865 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:43.532283 (+ 418us) server_negotiation.cc:299] Negotiation successful
0807 06:26:43.532401 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
I20260807 06:26:43.533113 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:43.526660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:36209 (local address 127.15.254.1:49525)
0807 06:26:43.527505 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:43.527523 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:43.527683 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:43.528188 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:43.528191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:43.528209 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:43.528367 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:43.528374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.529412 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.529415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:43.530900 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:43.530918 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.532003 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.532009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:43.532134 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:43.532170 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 06:26:43.532801 (+ 631us) client_negotiation.cc:240] Negotiation successful
0807 06:26:43.532903 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":341,"thread_start_us":99,"threads_started":1}
I20260807 06:26:43.534597 9581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "727156c4134647f394cd45f70824b436" candidate_uuid: "02733a436b1e44caa380acca6f848a86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e51a727d774f4e9193013fa6b010d10d" is_pre_election: true
I20260807 06:26:43.535014 9581 raft_consensus.cc:2454] T 727156c4134647f394cd45f70824b436 P e51a727d774f4e9193013fa6b010d10d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02733a436b1e44caa380acca6f848a86 in term 0.
I20260807 06:26:43.535310 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:43.526870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:33043 (local address 127.15.254.2:38335)
0807 06:26:43.527101 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:43.527109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:43.527345 (+ 236us) server_negotiation.cc:407] Connection header received
0807 06:26:43.527555 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:43.527560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:43.527628 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:43.527706 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:43.528346 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.529212 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.531038 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.531672 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.531809 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:43.531869 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:43.535065 (+ 3196us) server_negotiation.cc:299] Negotiation successful
0807 06:26:43.535173 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260807 06:26:43.535840 9128 leader_election.cc:302] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02733a436b1e44caa380acca6f848a86, e51a727d774f4e9193013fa6b010d10d; no voters:
I20260807 06:26:43.536103 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:43.526660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:38335 (local address 127.15.254.1:33043)
0807 06:26:43.527236 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:43.527248 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:43.527422 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:43.527955 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:43.527960 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:43.528007 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:43.528183 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:43.528191 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.529379 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.529385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:43.530881 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:43.530896 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.534624 (+ 3728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.534629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:43.534865 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:43.534895 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 06:26:43.535818 (+ 923us) client_negotiation.cc:240] Negotiation successful
0807 06:26:43.535893 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":4,"thread_start_us":163,"threads_started":1}
I20260807 06:26:43.536176 9671 raft_consensus.cc:2781] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:26:43.536289 9671 raft_consensus.cc:482] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:26:43.536348 9671 raft_consensus.cc:3037] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:43.536844 9397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "727156c4134647f394cd45f70824b436" candidate_uuid: "02733a436b1e44caa380acca6f848a86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce5cbe7e1be49eaabca0fb8bd284728" is_pre_election: true
I20260807 06:26:43.537137 9397 raft_consensus.cc:2454] T 727156c4134647f394cd45f70824b436 P bce5cbe7e1be49eaabca0fb8bd284728 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02733a436b1e44caa380acca6f848a86 in term 0.
I20260807 06:26:43.537806 9671 raft_consensus.cc:504] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce5cbe7e1be49eaabca0fb8bd284728" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38335 } } peers { permanent_uuid: "e51a727d774f4e9193013fa6b010d10d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36209 } } peers { permanent_uuid: "02733a436b1e44caa380acca6f848a86" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42073 } }
I20260807 06:26:43.538139 9671 leader_election.cc:288] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86 [CANDIDATE]: Term 1 election: Requested vote from peers bce5cbe7e1be49eaabca0fb8bd284728 (127.15.254.2:38335), e51a727d774f4e9193013fa6b010d10d (127.15.254.3:36209)
I20260807 06:26:43.538570 9397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "727156c4134647f394cd45f70824b436" candidate_uuid: "02733a436b1e44caa380acca6f848a86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce5cbe7e1be49eaabca0fb8bd284728"
I20260807 06:26:43.538582 9581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "727156c4134647f394cd45f70824b436" candidate_uuid: "02733a436b1e44caa380acca6f848a86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e51a727d774f4e9193013fa6b010d10d"
I20260807 06:26:43.538705 9397 raft_consensus.cc:3037] T 727156c4134647f394cd45f70824b436 P bce5cbe7e1be49eaabca0fb8bd284728 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:43.538705 9581 raft_consensus.cc:3037] T 727156c4134647f394cd45f70824b436 P e51a727d774f4e9193013fa6b010d10d [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:43.540258 9397 raft_consensus.cc:2454] T 727156c4134647f394cd45f70824b436 P bce5cbe7e1be49eaabca0fb8bd284728 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02733a436b1e44caa380acca6f848a86 in term 1.
I20260807 06:26:43.540258 9581 raft_consensus.cc:2454] T 727156c4134647f394cd45f70824b436 P e51a727d774f4e9193013fa6b010d10d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02733a436b1e44caa380acca6f848a86 in term 1.
I20260807 06:26:43.540616 9128 leader_election.cc:302] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02733a436b1e44caa380acca6f848a86, e51a727d774f4e9193013fa6b010d10d; no voters:
I20260807 06:26:43.540822 9671 raft_consensus.cc:2781] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:26:43.541122 9671 raft_consensus.cc:686] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86 [term 1 LEADER]: Becoming Leader. State: Replica: 02733a436b1e44caa380acca6f848a86, State: Running, Role: LEADER
I20260807 06:26:43.541502 9671 consensus_queue.cc:237] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce5cbe7e1be49eaabca0fb8bd284728" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38335 } } peers { permanent_uuid: "e51a727d774f4e9193013fa6b010d10d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36209 } } peers { permanent_uuid: "02733a436b1e44caa380acca6f848a86" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42073 } }
I20260807 06:26:43.545166 9018 catalog_manager.cc:5495] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02733a436b1e44caa380acca6f848a86 (127.15.254.1). New cstate: current_term: 1 leader_uuid: "02733a436b1e44caa380acca6f848a86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce5cbe7e1be49eaabca0fb8bd284728" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e51a727d774f4e9193013fa6b010d10d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02733a436b1e44caa380acca6f848a86" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 42073 } health_report { overall_health: HEALTHY } } }
I20260807 06:26:43.629274 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:43.624849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33844 (local address 127.15.254.1:42073)
0807 06:26:43.625078 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:43.625083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:43.625102 (+ 19us) server_negotiation.cc:407] Connection header received
0807 06:26:43.625229 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:43.625232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:43.625354 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:43.625473 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:26:43.625721 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.626278 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.627369 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:43.627608 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:43.627705 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:43.628065 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:26:43.628569 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:26:43.628680 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:43.629002 (+ 322us) server_negotiation.cc:299] Negotiation successful
0807 06:26:43.629087 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 06:26:43.636127 9581 raft_consensus.cc:1261] T 727156c4134647f394cd45f70824b436 P e51a727d774f4e9193013fa6b010d10d [term 1 FOLLOWER]: Refusing update from remote peer 02733a436b1e44caa380acca6f848a86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:26:43.636149 9397 raft_consensus.cc:1261] T 727156c4134647f394cd45f70824b436 P bce5cbe7e1be49eaabca0fb8bd284728 [term 1 FOLLOWER]: Refusing update from remote peer 02733a436b1e44caa380acca6f848a86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:26:43.636968 9677 consensus_queue.cc:1035] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a727d774f4e9193013fa6b010d10d" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:26:43.637192 9671 consensus_queue.cc:1035] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce5cbe7e1be49eaabca0fb8bd284728" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:26:43.644564 9680 mvcc.cc:205] Tried to move back new op lower bound from 7315800078885376000 to 7315800078508593152. Current Snapshot: MvccSnapshot[applied={T|T < 7315800078885376000}]
I20260807 06:26:43.644749 9682 mvcc.cc:205] Tried to move back new op lower bound from 7315800078885376000 to 7315800078508593152. Current Snapshot: MvccSnapshot[applied={T|T < 7315800078885376000}]
I20260807 06:26:43.646203 9681 mvcc.cc:205] Tried to move back new op lower bound from 7315800078885376000 to 7315800078508593152. Current Snapshot: MvccSnapshot[applied={T|T < 7315800078885376000}]
I20260807 06:26:43.655324 9021 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:26:43.655535 9021 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:26:43.658329 9021 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2d3ad9cad0e4df9a68887b9838f10c6: Sending DeleteTablet for 3 replicas of tablet 727156c4134647f394cd45f70824b436
I20260807 06:26:43.659190 9369 tablet_service.cc:1514] Processing DeleteTablet for tablet 727156c4134647f394cd45f70824b436 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:26:43 UTC) from {username='slave'} at 127.0.0.1:56062
I20260807 06:26:43.659235 9192 tablet_service.cc:1514] Processing DeleteTablet for tablet 727156c4134647f394cd45f70824b436 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:26:43 UTC) from {username='slave'} at 127.0.0.1:33834
I20260807 06:26:43.659266 9545 tablet_service.cc:1514] Processing DeleteTablet for tablet 727156c4134647f394cd45f70824b436 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:26:43 UTC) from {username='slave'} at 127.0.0.1:47144
I20260807 06:26:43.659936 9692 tablet_replica.cc:330] T 727156c4134647f394cd45f70824b436 P bce5cbe7e1be49eaabca0fb8bd284728: stopping tablet replica
I20260807 06:26:43.659972 9690 tablet_replica.cc:330] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86: stopping tablet replica
I20260807 06:26:43.660311 9692 raft_consensus.cc:2229] T 727156c4134647f394cd45f70824b436 P bce5cbe7e1be49eaabca0fb8bd284728 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:26:43.660514 9690 raft_consensus.cc:2229] T 727156c4134647f394cd45f70824b436 P 02733a436b1e44caa380acca6f848a86 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:26:43.660717 9692 raft_consensus.cc:2258] T 727156c4134647f394cd45f70824b436 P bce5cbe7e1be49eaabca0fb8bd284728 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:26:43.660928 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 9105
I20260807 06:26:43.661127 9693 tablet_replica.cc:330] T 727156c4134647f394cd45f70824b436 P e51a727d774f4e9193013fa6b010d10d: stopping tablet replica
I20260807 06:26:43.661504 9693 raft_consensus.cc:2229] T 727156c4134647f394cd45f70824b436 P e51a727d774f4e9193013fa6b010d10d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:26:43.661798 9693 raft_consensus.cc:2258] T 727156c4134647f394cd45f70824b436 P e51a727d774f4e9193013fa6b010d10d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:26:43.670213 9692 ts_tablet_manager.cc:1905] T 727156c4134647f394cd45f70824b436 P bce5cbe7e1be49eaabca0fb8bd284728: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:26:43.671145 9693 ts_tablet_manager.cc:1905] T 727156c4134647f394cd45f70824b436 P e51a727d774f4e9193013fa6b010d10d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:26:43.673578 9693 ts_tablet_manager.cc:1918] T 727156c4134647f394cd45f70824b436 P e51a727d774f4e9193013fa6b010d10d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:26:43.673713 9693 log.cc:1196] T 727156c4134647f394cd45f70824b436 P e51a727d774f4e9193013fa6b010d10d: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/727156c4134647f394cd45f70824b436
I20260807 06:26:43.674014 9693 ts_tablet_manager.cc:1939] T 727156c4134647f394cd45f70824b436 P e51a727d774f4e9193013fa6b010d10d: Deleting consensus metadata
I20260807 06:26:43.674265 9692 ts_tablet_manager.cc:1918] T 727156c4134647f394cd45f70824b436 P bce5cbe7e1be49eaabca0fb8bd284728: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:26:43.674419 9692 log.cc:1196] T 727156c4134647f394cd45f70824b436 P bce5cbe7e1be49eaabca0fb8bd284728: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/727156c4134647f394cd45f70824b436
I20260807 06:26:43.674817 9692 ts_tablet_manager.cc:1939] T 727156c4134647f394cd45f70824b436 P bce5cbe7e1be49eaabca0fb8bd284728: Deleting consensus metadata
I20260807 06:26:43.675621 8997 catalog_manager.cc:4841] TS e51a727d774f4e9193013fa6b010d10d (127.15.254.3:36209): tablet 727156c4134647f394cd45f70824b436 (table test-table [id=856cc30942094f6494560eb83ecf7528]) successfully deleted
I20260807 06:26:43.676158 8996 catalog_manager.cc:4841] TS bce5cbe7e1be49eaabca0fb8bd284728 (127.15.254.2:38335): tablet 727156c4134647f394cd45f70824b436 (table test-table [id=856cc30942094f6494560eb83ecf7528]) successfully deleted
W20260807 06:26:43.677596 9001 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.254.1:42073 (error 108)
I20260807 06:26:43.677920 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 9290
I20260807 06:26:43.684665 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:43.684137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:42073 (local address 127.0.0.1:33860)
0807 06:26:43.684329 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:43.684438 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.1:42073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260807 06:26:43.687701 9001 catalog_manager.cc:4584] TS 02733a436b1e44caa380acca6f848a86 (127.15.254.1:42073): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 727156c4134647f394cd45f70824b436: Network error: Client connection negotiation failed: client connection to 127.15.254.1:42073: connect: Connection refused (error 111)
I20260807 06:26:43.695032 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 9467
I20260807 06:26:43.707043 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 8977
2026-08-07T06:26:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1429 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:26:43 dist-test-slave-44d2 krb5kdc[9698](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:26:43 dist-test-slave-44d2 krb5kdc[9698](info): set up 2 sockets
Aug 07 06:26:43 dist-test-slave-44d2 krb5kdc[9698](info): commencing operation
krb5kdc: starting...
W20260807 06:26:44.938205 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.203s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:26:45 dist-test-slave-44d2 krb5kdc[9698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:26:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:26:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:26:45.078262 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:40313
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:40313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:26:45.192874 9716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:45.193176 9716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:45.193248 9716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:45.197429 9716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:26:45.197516 9716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:45.197554 9716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:45.197589 9716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:26:45.197626 9716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:26:45.202948 9716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:40313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:40313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:45.204104 9716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:45.205235 9716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:45.214082 9723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:45.214102 9725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:45.214241 9716 server_base.cc:1034] running on GCE node
W20260807 06:26:45.214134 9722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:45.215062 9716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:45.216150 9716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:45.217329 9716 hybrid_clock.cc:648] HybridClock initialized: now 1786084005217302 us; error 42 us; skew 500 ppm
Aug 07 06:26:45 dist-test-slave-44d2 krb5kdc[9698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084005, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:26:45.224689 9716 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:26:45.225739 9716 webserver.cc:460] Webserver started at http://127.15.254.62:45999/ using document root <none> and password file <none>
I20260807 06:26:45.226337 9716 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:45.226435 9716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:45.226676 9716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:45.228892 9716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "bead17ee879344dcb976054785208c94"
format_stamp: "Formatted at 2026-08-07 06:26:45 on dist-test-slave-44d2"
server_key: "3635817218fa13c9f4f77813d3c6f154"
server_key_iv: "514aac98e1a272d8056fecd1697744d1"
server_key_version: "encryptionkey@0"
I20260807 06:26:45.229498 9716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "bead17ee879344dcb976054785208c94"
format_stamp: "Formatted at 2026-08-07 06:26:45 on dist-test-slave-44d2"
server_key: "3635817218fa13c9f4f77813d3c6f154"
server_key_iv: "514aac98e1a272d8056fecd1697744d1"
server_key_version: "encryptionkey@0"
I20260807 06:26:45.233520 9716 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:26:45.236367 9735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:45.237746 9716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:26:45.237946 9716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "bead17ee879344dcb976054785208c94"
format_stamp: "Formatted at 2026-08-07 06:26:45 on dist-test-slave-44d2"
server_key: "3635817218fa13c9f4f77813d3c6f154"
server_key_iv: "514aac98e1a272d8056fecd1697744d1"
server_key_version: "encryptionkey@0"
I20260807 06:26:45.238075 9716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:45.258013 9716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:45.261668 9716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:45.261973 9716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:45.270233 9716 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:40313
I20260807 06:26:45.270257 9804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:40313 every 8 connection(s)
I20260807 06:26:45.271284 9716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:26:45.274369 9805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:45.276427 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 9716
I20260807 06:26:45.276589 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:26:45.276919 16376 external_mini_cluster.cc:1428] Setting key 1c1fab5832d039e3dedd5239f9ecdb7e
I20260807 06:26:45.281872 9805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bead17ee879344dcb976054785208c94: Bootstrap starting.
I20260807 06:26:45.284677 9805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bead17ee879344dcb976054785208c94: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:45.285494 9805 log.cc:824] T 00000000000000000000000000000000 P bead17ee879344dcb976054785208c94: Log is configured to *not* fsync() on all Append() calls
Aug 07 06:26:45 dist-test-slave-44d2 krb5kdc[9698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:26:45.287738 9805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bead17ee879344dcb976054785208c94: No bootstrap required, opened a new log
I20260807 06:26:45.292613 9805 raft_consensus.cc:348] T 00000000000000000000000000000000 P bead17ee879344dcb976054785208c94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bead17ee879344dcb976054785208c94" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 40313 } }
I20260807 06:26:45.292901 9805 raft_consensus.cc:374] T 00000000000000000000000000000000 P bead17ee879344dcb976054785208c94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:45.292971 9805 raft_consensus.cc:729] T 00000000000000000000000000000000 P bead17ee879344dcb976054785208c94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bead17ee879344dcb976054785208c94, State: Initialized, Role: FOLLOWER
I20260807 06:26:45.293517 9805 consensus_queue.cc:260] T 00000000000000000000000000000000 P bead17ee879344dcb976054785208c94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bead17ee879344dcb976054785208c94" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 40313 } }
I20260807 06:26:45.293723 9805 raft_consensus.cc:388] T 00000000000000000000000000000000 P bead17ee879344dcb976054785208c94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:26:45.293803 9805 raft_consensus.cc:482] T 00000000000000000000000000000000 P bead17ee879344dcb976054785208c94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:26:45.294116 9805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bead17ee879344dcb976054785208c94 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:45.294405 9809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:45.278502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32824 (local address 127.15.254.62:40313)
0807 06:26:45.278960 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:45.278986 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:45.279030 (+ 44us) server_negotiation.cc:407] Connection header received
0807 06:26:45.279854 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:45.279880 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:45.280316 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:45.280749 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:45.281489 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:45.282933 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:45.283872 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:45.284614 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:45.284802 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:45.288098 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:45.288106 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:45.288127 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:45.288162 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:45.290498 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:45.291042 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:45.291048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:45.291055 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:45.291199 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:45.291485 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:45.291488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:45.291490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:45.292024 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:45.292232 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:45.292459 (+ 227us) server_negotiation.cc:299] Negotiation successful
0807 06:26:45.292803 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":119,"threads_started":1}
I20260807 06:26:45.295275 9805 raft_consensus.cc:504] T 00000000000000000000000000000000 P bead17ee879344dcb976054785208c94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bead17ee879344dcb976054785208c94" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 40313 } }
I20260807 06:26:45.295643 9805 leader_election.cc:302] T 00000000000000000000000000000000 P bead17ee879344dcb976054785208c94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bead17ee879344dcb976054785208c94; no voters:
I20260807 06:26:45.295991 9805 leader_election.cc:288] T 00000000000000000000000000000000 P bead17ee879344dcb976054785208c94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:26:45.296100 9812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bead17ee879344dcb976054785208c94 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:26:45.296303 9812 raft_consensus.cc:686] T 00000000000000000000000000000000 P bead17ee879344dcb976054785208c94 [term 1 LEADER]: Becoming Leader. State: Replica: bead17ee879344dcb976054785208c94, State: Running, Role: LEADER
I20260807 06:26:45.296850 9812 consensus_queue.cc:237] T 00000000000000000000000000000000 P bead17ee879344dcb976054785208c94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bead17ee879344dcb976054785208c94" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 40313 } }
I20260807 06:26:45.297426 9805 sys_catalog.cc:564] T 00000000000000000000000000000000 P bead17ee879344dcb976054785208c94 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:26:45.298655 9815 sys_catalog.cc:455] T 00000000000000000000000000000000 P bead17ee879344dcb976054785208c94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bead17ee879344dcb976054785208c94. Latest consensus state: current_term: 1 leader_uuid: "bead17ee879344dcb976054785208c94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bead17ee879344dcb976054785208c94" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 40313 } } }
I20260807 06:26:45.298801 9815 sys_catalog.cc:458] T 00000000000000000000000000000000 P bead17ee879344dcb976054785208c94 [sys.catalog]: This master's current role is: LEADER
I20260807 06:26:45.298863 9813 sys_catalog.cc:455] T 00000000000000000000000000000000 P bead17ee879344dcb976054785208c94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bead17ee879344dcb976054785208c94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bead17ee879344dcb976054785208c94" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 40313 } } }
I20260807 06:26:45.299000 9813 sys_catalog.cc:458] T 00000000000000000000000000000000 P bead17ee879344dcb976054785208c94 [sys.catalog]: This master's current role is: LEADER
W20260807 06:26:45.303056 9835 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bead17ee879344dcb976054785208c94: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:26:45.303150 9835 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:26:45.303256 9822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:26:45.304198 9822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:26:45.310436 9822 catalog_manager.cc:1347] Generated new cluster ID: fd65d681c0564a9d869e4ab30bb9f045
I20260807 06:26:45.310514 9822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:26:45.338397 9822 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:26:45.339485 9822 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:26:45.346124 9822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bead17ee879344dcb976054785208c94: Generated new TSK 0
I20260807 06:26:45.346961 9822 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
I20260807 06:26:45.387758 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:0
--local_ip_for_outbound_sockets=127.15.254.1
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:40313
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:26:45.506765 9844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:45.507051 9844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:45.507095 9844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:45.511361 9844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:45.511432 9844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:45.511528 9844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:26:45.517815 9844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:40313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:45.519006 9844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:45.520092 9844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:45.530373 9850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:45.530380 9849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:45.530402 9852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:45.530794 9844 server_base.cc:1034] running on GCE node
I20260807 06:26:45.531163 9844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:45.531797 9844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:45.532994 9844 hybrid_clock.cc:648] HybridClock initialized: now 1786084005532975 us; error 33 us; skew 500 ppm
Aug 07 06:26:45 dist-test-slave-44d2 krb5kdc[9698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084005, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:26:45.536027 9844 init.cc:377] Logged in from keytab as kudu/127.15.254.1@KRBTEST.COM (short username kudu)
I20260807 06:26:45.537097 9844 webserver.cc:460] Webserver started at http://127.15.254.1:45785/ using document root <none> and password file <none>
I20260807 06:26:45.537662 9844 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:45.537725 9844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:45.537925 9844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:45.539983 9844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-0/data/instance:
uuid: "693a9c101b7c4ee9b99d9df31d5c2bb7"
format_stamp: "Formatted at 2026-08-07 06:26:45 on dist-test-slave-44d2"
server_key: "f122da2b06a1ad3b2a11197c6c94d4f7"
server_key_iv: "cff448866b3eb9210ddc34ca4f59b0bf"
server_key_version: "encryptionkey@0"
I20260807 06:26:45.540580 9844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance:
uuid: "693a9c101b7c4ee9b99d9df31d5c2bb7"
format_stamp: "Formatted at 2026-08-07 06:26:45 on dist-test-slave-44d2"
server_key: "f122da2b06a1ad3b2a11197c6c94d4f7"
server_key_iv: "cff448866b3eb9210ddc34ca4f59b0bf"
server_key_version: "encryptionkey@0"
I20260807 06:26:45.544566 9844 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 06:26:45.547233 9863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:45.548699 9844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 06:26:45.548839 9844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "693a9c101b7c4ee9b99d9df31d5c2bb7"
format_stamp: "Formatted at 2026-08-07 06:26:45 on dist-test-slave-44d2"
server_key: "f122da2b06a1ad3b2a11197c6c94d4f7"
server_key_iv: "cff448866b3eb9210ddc34ca4f59b0bf"
server_key_version: "encryptionkey@0"
I20260807 06:26:45.548947 9844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:45.566139 9844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:45.569589 9844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:45.569800 9844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:45.570451 9844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:26:45.571307 9844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:26:45.571357 9844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:45.571434 9844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:26:45.571467 9844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:45.581354 9844 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:41103
I20260807 06:26:45.581390 10015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:41103 every 8 connection(s)
I20260807 06:26:45.582273 9844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:26:45.585577 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 9844
I20260807 06:26:45.585727 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance
I20260807 06:26:45.586047 16376 external_mini_cluster.cc:1428] Setting key db08f0012c8b8711003b335646befedd
Aug 07 06:26:45 dist-test-slave-44d2 krb5kdc[9698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084005, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:26:45.596772 9809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:45.584097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:41037 (local address 127.15.254.62:40313)
0807 06:26:45.584239 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:45.584243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:45.585405 (+ 1162us) server_negotiation.cc:407] Connection header received
0807 06:26:45.586482 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:45.586485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:45.586538 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:45.586619 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:45.588209 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:45.588811 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:45.590016 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:45.590248 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:45.590361 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:45.593029 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:45.593036 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:45.593042 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:45.593081 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:45.594657 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:45.595181 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:45.595186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:45.595188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:45.595271 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:45.595638 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:45.595641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:45.595643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:45.595827 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:45.595936 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:45.596442 (+ 506us) server_negotiation.cc:299] Negotiation successful
0807 06:26:45.596594 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for kudu/127.15.254.2@KRBTEST.COM; defaulting to no policy
I20260807 06:26:45.597821 10019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:45.584466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:40313 (local address 127.15.254.1:41037)
0807 06:26:45.585232 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:45.585277 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:45.586257 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:45.586998 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:45.587006 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:45.587506 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:26:45.588058 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:45.588073 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:45.588974 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:45.588979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:45.589877 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:45.589887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:45.590407 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:45.590411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:45.590590 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:45.591241 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:26:45.591257 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:26:45.592791 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:26:45.594782 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:45.594789 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:45.594805 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:45.595078 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:45.595362 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:45.595368 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:45.595373 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:45.595539 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:45.595936 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:26:45.595943 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:26:45.596186 (+ 243us) client_negotiation.cc:769] Sending connection context
0807 06:26:45.596383 (+ 197us) client_negotiation.cc:240] Negotiation successful
0807 06:26:45.596752 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":111,"threads_started":1}
I20260807 06:26:45.599218 10017 heartbeater.cc:347] Connected to a master server at 127.15.254.62:40313
I20260807 06:26:45.599536 10017 heartbeater.cc:464] Registering TS with master...
I20260807 06:26:45.600142 10017 heartbeater.cc:511] Master 127.15.254.62:40313 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.2@KRBTEST.COM" created.
I20260807 06:26:45.601897 9755 ts_manager.cc:194] Registered new tserver with Master: 693a9c101b7c4ee9b99d9df31d5c2bb7 (127.15.254.1:41103)
I20260807 06:26:45.603050 9755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.1@KRBTEST.COM'} at 127.15.254.1:41037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
I20260807 06:26:45.620361 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:0
--local_ip_for_outbound_sockets=127.15.254.2
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:40313
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:26:45.737830 10022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:45.738102 10022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:45.738145 10022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:45.742286 10022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:45.742343 10022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:45.742443 10022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:26:45.747776 10022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:40313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:45.748978 10022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:45.750051 10022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:45.760342 10032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:45.760406 10029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:45.760406 10030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:45.760669 10022 server_base.cc:1034] running on GCE node
I20260807 06:26:45.761092 10022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:45.761716 10022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:45.762873 10022 hybrid_clock.cc:648] HybridClock initialized: now 1786084005762836 us; error 54 us; skew 500 ppm
Aug 07 06:26:45 dist-test-slave-44d2 krb5kdc[9698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084005, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:26:45.766005 10022 init.cc:377] Logged in from keytab as kudu/127.15.254.2@KRBTEST.COM (short username kudu)
I20260807 06:26:45.766896 10022 webserver.cc:460] Webserver started at http://127.15.254.2:41417/ using document root <none> and password file <none>
I20260807 06:26:45.767530 10022 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:45.767596 10022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:45.767814 10022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:45.769989 10022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-1/data/instance:
uuid: "010b4a8232d24694a74144ef8be615ee"
format_stamp: "Formatted at 2026-08-07 06:26:45 on dist-test-slave-44d2"
server_key: "4641a57e192c69b545f14432e52926c8"
server_key_iv: "e0a5dd99041c370687f76a92658bf204"
server_key_version: "encryptionkey@0"
I20260807 06:26:45.770565 10022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance:
uuid: "010b4a8232d24694a74144ef8be615ee"
format_stamp: "Formatted at 2026-08-07 06:26:45 on dist-test-slave-44d2"
server_key: "4641a57e192c69b545f14432e52926c8"
server_key_iv: "e0a5dd99041c370687f76a92658bf204"
server_key_version: "encryptionkey@0"
I20260807 06:26:45.774478 10022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:26:45.777081 10042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:45.778347 10022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 06:26:45.778473 10022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "010b4a8232d24694a74144ef8be615ee"
format_stamp: "Formatted at 2026-08-07 06:26:45 on dist-test-slave-44d2"
server_key: "4641a57e192c69b545f14432e52926c8"
server_key_iv: "e0a5dd99041c370687f76a92658bf204"
server_key_version: "encryptionkey@0"
I20260807 06:26:45.778582 10022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:45.792580 10022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:45.795806 10022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:45.796011 10022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:45.796690 10022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:26:45.797765 10022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:26:45.797828 10022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:45.797894 10022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:26:45.797917 10022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:45.807811 10022 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:38955
I20260807 06:26:45.807897 10209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:38955 every 8 connection(s)
I20260807 06:26:45.808872 10022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:26:45.818081 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 10022
I20260807 06:26:45.818233 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance
I20260807 06:26:45.818603 16376 external_mini_cluster.cc:1428] Setting key 6c6b8f543306439f6fdb6e18cf030ce2
Aug 07 06:26:45 dist-test-slave-44d2 krb5kdc[9698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084005, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:26:45.825801 9809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:45.811003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:53287 (local address 127.15.254.62:40313)
0807 06:26:45.811169 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:45.811174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:45.812459 (+ 1285us) server_negotiation.cc:407] Connection header received
0807 06:26:45.813540 (+ 1081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:45.813543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:45.813594 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:45.813687 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:45.814979 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:45.815503 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:45.816847 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:45.817080 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:45.817166 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:45.820376 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:45.820380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:45.820383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:45.820414 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:45.823218 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:45.824002 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:45.824006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:45.824007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:45.824101 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:45.824562 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:45.824566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:45.824567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:45.824772 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:45.824853 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:45.825560 (+ 707us) server_negotiation.cc:299] Negotiation successful
0807 06:26:45.825675 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 06:26:45.827128 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:45.811440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:40313 (local address 127.15.254.2:53287)
0807 06:26:45.812296 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:45.812341 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:45.813305 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:45.813854 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:45.813861 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:45.814302 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:26:45.814826 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:45.814840 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:45.815624 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:45.815627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:45.816716 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:45.816727 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:45.817231 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:45.817234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:45.817396 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:45.818007 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:26:45.818031 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:26:45.820187 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:26:45.823363 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:45.823373 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:45.823393 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:45.823819 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:45.824204 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:45.824208 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:45.824210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:45.824422 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:45.824893 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:26:45.824903 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:26:45.825262 (+ 359us) client_negotiation.cc:769] Sending connection context
0807 06:26:45.825483 (+ 221us) client_negotiation.cc:240] Negotiation successful
0807 06:26:45.825900 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":125,"threads_started":1}
I20260807 06:26:45.828594 10210 heartbeater.cc:347] Connected to a master server at 127.15.254.62:40313
I20260807 06:26:45.828878 10210 heartbeater.cc:464] Registering TS with master...
I20260807 06:26:45.829419 10210 heartbeater.cc:511] Master 127.15.254.62:40313 requested a full tablet report, sending...
I20260807 06:26:45.830854 9755 ts_manager.cc:194] Registered new tserver with Master: 010b4a8232d24694a74144ef8be615ee (127.15.254.2:38955)
WARNING: no policy specified for kudu/127.15.254.3@KRBTEST.COM; defaulting to no policy
I20260807 06:26:45.831744 9755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.2@KRBTEST.COM'} at 127.15.254.2:53287
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
I20260807 06:26:45.852952 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:0
--local_ip_for_outbound_sockets=127.15.254.3
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:40313
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:26:45.970707 10215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:45.970978 10215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:45.971022 10215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:45.975104 10215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:45.975167 10215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:45.975224 10215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:26:45.980521 10215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:40313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:45.981729 10215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:45.982868 10215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:45.993259 10221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:45.993220 10225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:45.993247 10223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:45.993597 10215 server_base.cc:1034] running on GCE node
I20260807 06:26:45.994138 10215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:45.994807 10215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:45.996156 10215 hybrid_clock.cc:648] HybridClock initialized: now 1786084005996130 us; error 40 us; skew 500 ppm
Aug 07 06:26:45 dist-test-slave-44d2 krb5kdc[9698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084005, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:26:45.999593 10215 init.cc:377] Logged in from keytab as kudu/127.15.254.3@KRBTEST.COM (short username kudu)
I20260807 06:26:46.000695 10215 webserver.cc:460] Webserver started at http://127.15.254.3:33597/ using document root <none> and password file <none>
I20260807 06:26:46.001341 10215 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:46.001451 10215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:46.001715 10215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:46.003990 10215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-2/data/instance:
uuid: "41b1ef9463764e4eaca0f44e150e0f3f"
format_stamp: "Formatted at 2026-08-07 06:26:45 on dist-test-slave-44d2"
server_key: "97a003b331260f03d342b07bc98364bf"
server_key_iv: "c757e7a105310cad3b044fb367af2d30"
server_key_version: "encryptionkey@0"
I20260807 06:26:46.004679 10215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance:
uuid: "41b1ef9463764e4eaca0f44e150e0f3f"
format_stamp: "Formatted at 2026-08-07 06:26:45 on dist-test-slave-44d2"
server_key: "97a003b331260f03d342b07bc98364bf"
server_key_iv: "c757e7a105310cad3b044fb367af2d30"
server_key_version: "encryptionkey@0"
I20260807 06:26:46.008996 10215 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260807 06:26:46.011826 10233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:46.013381 10215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:26:46.013533 10215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "41b1ef9463764e4eaca0f44e150e0f3f"
format_stamp: "Formatted at 2026-08-07 06:26:45 on dist-test-slave-44d2"
server_key: "97a003b331260f03d342b07bc98364bf"
server_key_iv: "c757e7a105310cad3b044fb367af2d30"
server_key_version: "encryptionkey@0"
I20260807 06:26:46.013653 10215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:46.048586 10215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:46.052289 10215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:46.052537 10215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:46.053345 10215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:26:46.054240 10215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:26:46.054323 10215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:46.054396 10215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:26:46.054441 10215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:46.064301 10215 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:41287
I20260807 06:26:46.064320 10388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:41287 every 8 connection(s)
I20260807 06:26:46.065291 10215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:26:46.071080 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 10215
I20260807 06:26:46.071251 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance
I20260807 06:26:46.071559 16376 external_mini_cluster.cc:1428] Setting key bd8a29991b0c2529f9689a51e3a94e95
Aug 07 06:26:46 dist-test-slave-44d2 krb5kdc[9698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084005, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:26:46.080574 9809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:46.067222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:46579 (local address 127.15.254.62:40313)
0807 06:26:46.067436 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:46.067440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:46.068561 (+ 1121us) server_negotiation.cc:407] Connection header received
0807 06:26:46.069652 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:46.069657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:46.069727 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:46.069853 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:46.071184 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.071930 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.073758 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.074019 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.074104 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:46.076844 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:46.076848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:46.076850 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:46.076875 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:46.078473 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:46.079038 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:46.079041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:46.079042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:46.079130 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:46.079456 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:46.079459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:46.079460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:46.079641 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:46.079732 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:46.080263 (+ 531us) server_negotiation.cc:299] Negotiation successful
0807 06:26:46.080379 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 06:26:46.081364 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:46.067556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:40313 (local address 127.15.254.3:46579)
0807 06:26:46.068370 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:46.068419 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:46.069404 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:46.070058 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:46.070067 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:46.070473 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:26:46.071011 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:46.071028 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.072081 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.072092 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:46.073605 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:46.073619 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.074173 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.074177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:46.074341 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:46.074922 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:26:46.074938 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:26:46.076662 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:26:46.078592 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:46.078597 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:46.078610 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:46.078874 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:46.079224 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:46.079227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:46.079228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:46.079365 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:46.079742 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:26:46.079746 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:26:46.079983 (+ 237us) client_negotiation.cc:769] Sending connection context
0807 06:26:46.080168 (+ 185us) client_negotiation.cc:240] Negotiation successful
0807 06:26:46.080433 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":122,"threads_started":1}
I20260807 06:26:46.082691 10390 heartbeater.cc:347] Connected to a master server at 127.15.254.62:40313
I20260807 06:26:46.083029 10390 heartbeater.cc:464] Registering TS with master...
I20260807 06:26:46.083632 10390 heartbeater.cc:511] Master 127.15.254.62:40313 requested a full tablet report, sending...
I20260807 06:26:46.084864 9755 ts_manager.cc:194] Registered new tserver with Master: 41b1ef9463764e4eaca0f44e150e0f3f (127.15.254.3:41287)
I20260807 06:26:46.085583 9755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.3@KRBTEST.COM'} at 127.15.254.3:46579
I20260807 06:26:46.086089 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:26:46.096421 9809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:46.088571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32836 (local address 127.15.254.62:40313)
0807 06:26:46.088692 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:46.088695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:46.088885 (+ 190us) server_negotiation.cc:407] Connection header received
0807 06:26:46.088982 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:46.088984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:46.089049 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:46.089119 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:46.089773 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.090272 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.091102 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.091310 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.091388 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:46.093138 (+ 1750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:46.093142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:46.093144 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:46.093170 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:46.094759 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:46.095249 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:46.095252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:46.095253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:46.095331 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:46.095608 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:46.095610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:46.095611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:46.095802 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:46.095932 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:46.096204 (+ 272us) server_negotiation.cc:299] Negotiation successful
0807 06:26:46.096285 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 06:26:46.100080 9755 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:26:46.103510 9755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:26:46.120157 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:46.113095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:41103 (local address 127.0.0.1:41682)
0807 06:26:46.114629 (+ 1534us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:46.114646 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:46.114743 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:46.115506 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:46.115509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:46.115533 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:46.115624 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:46.115632 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.117334 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.117337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:46.118526 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:46.118534 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.119576 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.119580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:46.119689 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:46.119730 (+ 41us) client_negotiation.cc:769] Sending connection context
0807 06:26:46.119850 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 06:26:46.119987 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1461,"mutex_wait_us":20,"thread_start_us":115,"threads_started":1}
I20260807 06:26:46.120168 10406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:46.113804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41682 (local address 127.15.254.1:41103)
0807 06:26:46.114133 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:46.114141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:46.115074 (+ 933us) server_negotiation.cc:407] Connection header received
0807 06:26:46.115176 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:46.115181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:46.115368 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:46.115524 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:46.115802 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.117192 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.118640 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.119451 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.119561 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:46.119706 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:46.119894 (+ 188us) server_negotiation.cc:299] Negotiation successful
0807 06:26:46.119985 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":119,"threads_started":1}
I20260807 06:26:46.120568 10403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:46.114222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:38955 (local address 127.0.0.1:56508)
0807 06:26:46.115028 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:46.115043 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:46.115147 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:46.115698 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:46.115701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:46.115718 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:46.115799 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:46.115804 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.117228 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.117231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:46.119111 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:46.119119 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.120113 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.120117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:46.120228 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:46.120255 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 06:26:46.120331 (+ 76us) client_negotiation.cc:240] Negotiation successful
0807 06:26:46.120409 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":3,"thread_start_us":118,"threads_started":1}
I20260807 06:26:46.120946 10409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:46.114506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56508 (local address 127.15.254.2:38955)
0807 06:26:46.115002 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:46.115014 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:46.115134 (+ 120us) server_negotiation.cc:407] Connection header received
0807 06:26:46.115275 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:46.115280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:46.115428 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:46.115556 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:46.115926 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.117062 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.119224 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.119987 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.120096 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:46.120405 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:46.120469 (+ 64us) server_negotiation.cc:299] Negotiation successful
0807 06:26:46.120600 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":122,"threads_started":1}
I20260807 06:26:46.122272 10404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:46.112984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:41287 (local address 127.0.0.1:57534)
0807 06:26:46.113573 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:46.113630 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:46.113786 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:46.115008 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:46.115015 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:46.115044 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:46.115234 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:46.115243 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.116905 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.116909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:46.118426 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:46.118438 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.121861 (+ 3423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.121865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:46.121969 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:46.121993 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 06:26:46.122078 (+ 85us) client_negotiation.cc:240] Negotiation successful
0807 06:26:46.122144 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":66,"threads_started":1}
I20260807 06:26:46.122372 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:46.113366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57534 (local address 127.15.254.3:41287)
0807 06:26:46.114094 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:46.114110 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:46.114139 (+ 29us) server_negotiation.cc:407] Connection header received
0807 06:26:46.114231 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:46.114237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:46.114502 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:46.114697 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:46.115364 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.116741 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.118550 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.119445 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.119561 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:46.119706 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:46.122128 (+ 2422us) server_negotiation.cc:299] Negotiation successful
0807 06:26:46.122233 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":121,"threads_started":1}
I20260807 06:26:46.123413 9926 tablet_service.cc:1467] Processing CreateTablet for tablet d0488b855abb4f0da1bd01e2a7b2c851 (DEFAULT_TABLE table=test-table [id=7c93e2db74d046ca8b5167e926efe484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:26:46.123519 10119 tablet_service.cc:1467] Processing CreateTablet for tablet d0488b855abb4f0da1bd01e2a7b2c851 (DEFAULT_TABLE table=test-table [id=7c93e2db74d046ca8b5167e926efe484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:26:46.124329 10119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0488b855abb4f0da1bd01e2a7b2c851. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:46.124330 9926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0488b855abb4f0da1bd01e2a7b2c851. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:46.126806 10304 tablet_service.cc:1467] Processing CreateTablet for tablet d0488b855abb4f0da1bd01e2a7b2c851 (DEFAULT_TABLE table=test-table [id=7c93e2db74d046ca8b5167e926efe484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:26:46.127828 10304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0488b855abb4f0da1bd01e2a7b2c851. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:46.130877 10411 tablet_bootstrap.cc:492] T d0488b855abb4f0da1bd01e2a7b2c851 P 010b4a8232d24694a74144ef8be615ee: Bootstrap starting.
I20260807 06:26:46.132983 10411 tablet_bootstrap.cc:654] T d0488b855abb4f0da1bd01e2a7b2c851 P 010b4a8232d24694a74144ef8be615ee: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:46.133939 10413 tablet_bootstrap.cc:492] T d0488b855abb4f0da1bd01e2a7b2c851 P 693a9c101b7c4ee9b99d9df31d5c2bb7: Bootstrap starting.
I20260807 06:26:46.134298 10411 log.cc:824] T d0488b855abb4f0da1bd01e2a7b2c851 P 010b4a8232d24694a74144ef8be615ee: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:46.136236 10413 tablet_bootstrap.cc:654] T d0488b855abb4f0da1bd01e2a7b2c851 P 693a9c101b7c4ee9b99d9df31d5c2bb7: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:46.136613 10411 tablet_bootstrap.cc:492] T d0488b855abb4f0da1bd01e2a7b2c851 P 010b4a8232d24694a74144ef8be615ee: No bootstrap required, opened a new log
I20260807 06:26:46.136729 10416 tablet_bootstrap.cc:492] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f: Bootstrap starting.
I20260807 06:26:46.136814 10411 ts_tablet_manager.cc:1397] T d0488b855abb4f0da1bd01e2a7b2c851 P 010b4a8232d24694a74144ef8be615ee: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260807 06:26:46.137182 10413 log.cc:824] T d0488b855abb4f0da1bd01e2a7b2c851 P 693a9c101b7c4ee9b99d9df31d5c2bb7: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:46.138700 10416 tablet_bootstrap.cc:654] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:46.139263 10413 tablet_bootstrap.cc:492] T d0488b855abb4f0da1bd01e2a7b2c851 P 693a9c101b7c4ee9b99d9df31d5c2bb7: No bootstrap required, opened a new log
I20260807 06:26:46.139463 10413 ts_tablet_manager.cc:1397] T d0488b855abb4f0da1bd01e2a7b2c851 P 693a9c101b7c4ee9b99d9df31d5c2bb7: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260807 06:26:46.139822 10416 log.cc:824] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:46.140769 10411 raft_consensus.cc:348] T d0488b855abb4f0da1bd01e2a7b2c851 P 010b4a8232d24694a74144ef8be615ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1ef9463764e4eaca0f44e150e0f3f" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 41287 } } peers { permanent_uuid: "693a9c101b7c4ee9b99d9df31d5c2bb7" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 41103 } } peers { permanent_uuid: "010b4a8232d24694a74144ef8be615ee" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38955 } }
I20260807 06:26:46.141065 10411 raft_consensus.cc:374] T d0488b855abb4f0da1bd01e2a7b2c851 P 010b4a8232d24694a74144ef8be615ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:46.141136 10411 raft_consensus.cc:729] T d0488b855abb4f0da1bd01e2a7b2c851 P 010b4a8232d24694a74144ef8be615ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 010b4a8232d24694a74144ef8be615ee, State: Initialized, Role: FOLLOWER
I20260807 06:26:46.141682 10411 consensus_queue.cc:260] T d0488b855abb4f0da1bd01e2a7b2c851 P 010b4a8232d24694a74144ef8be615ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1ef9463764e4eaca0f44e150e0f3f" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 41287 } } peers { permanent_uuid: "693a9c101b7c4ee9b99d9df31d5c2bb7" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 41103 } } peers { permanent_uuid: "010b4a8232d24694a74144ef8be615ee" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38955 } }
I20260807 06:26:46.141821 10416 tablet_bootstrap.cc:492] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f: No bootstrap required, opened a new log
I20260807 06:26:46.142017 10416 ts_tablet_manager.cc:1397] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260807 06:26:46.142457 10210 heartbeater.cc:503] Master 127.15.254.62:40313 was elected leader, sending a full tablet report...
I20260807 06:26:46.142702 10411 ts_tablet_manager.cc:1428] T d0488b855abb4f0da1bd01e2a7b2c851 P 010b4a8232d24694a74144ef8be615ee: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260807 06:26:46.143918 10413 raft_consensus.cc:348] T d0488b855abb4f0da1bd01e2a7b2c851 P 693a9c101b7c4ee9b99d9df31d5c2bb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1ef9463764e4eaca0f44e150e0f3f" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 41287 } } peers { permanent_uuid: "693a9c101b7c4ee9b99d9df31d5c2bb7" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 41103 } } peers { permanent_uuid: "010b4a8232d24694a74144ef8be615ee" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38955 } }
I20260807 06:26:46.144328 10413 raft_consensus.cc:374] T d0488b855abb4f0da1bd01e2a7b2c851 P 693a9c101b7c4ee9b99d9df31d5c2bb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:46.144412 10413 raft_consensus.cc:729] T d0488b855abb4f0da1bd01e2a7b2c851 P 693a9c101b7c4ee9b99d9df31d5c2bb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 693a9c101b7c4ee9b99d9df31d5c2bb7, State: Initialized, Role: FOLLOWER
I20260807 06:26:46.144903 10413 consensus_queue.cc:260] T d0488b855abb4f0da1bd01e2a7b2c851 P 693a9c101b7c4ee9b99d9df31d5c2bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1ef9463764e4eaca0f44e150e0f3f" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 41287 } } peers { permanent_uuid: "693a9c101b7c4ee9b99d9df31d5c2bb7" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 41103 } } peers { permanent_uuid: "010b4a8232d24694a74144ef8be615ee" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38955 } }
I20260807 06:26:46.145645 10017 heartbeater.cc:503] Master 127.15.254.62:40313 was elected leader, sending a full tablet report...
I20260807 06:26:46.145864 10413 ts_tablet_manager.cc:1428] T d0488b855abb4f0da1bd01e2a7b2c851 P 693a9c101b7c4ee9b99d9df31d5c2bb7: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260807 06:26:46.146298 10416 raft_consensus.cc:348] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1ef9463764e4eaca0f44e150e0f3f" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 41287 } } peers { permanent_uuid: "693a9c101b7c4ee9b99d9df31d5c2bb7" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 41103 } } peers { permanent_uuid: "010b4a8232d24694a74144ef8be615ee" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38955 } }
I20260807 06:26:46.146587 10416 raft_consensus.cc:374] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:46.146657 10416 raft_consensus.cc:729] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41b1ef9463764e4eaca0f44e150e0f3f, State: Initialized, Role: FOLLOWER
I20260807 06:26:46.147089 10416 consensus_queue.cc:260] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1ef9463764e4eaca0f44e150e0f3f" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 41287 } } peers { permanent_uuid: "693a9c101b7c4ee9b99d9df31d5c2bb7" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 41103 } } peers { permanent_uuid: "010b4a8232d24694a74144ef8be615ee" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38955 } }
I20260807 06:26:46.147842 10390 heartbeater.cc:503] Master 127.15.254.62:40313 was elected leader, sending a full tablet report...
I20260807 06:26:46.148149 10416 ts_tablet_manager.cc:1428] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260807 06:26:46.161307 10424 raft_consensus.cc:482] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:26:46.161515 10424 raft_consensus.cc:504] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1ef9463764e4eaca0f44e150e0f3f" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 41287 } } peers { permanent_uuid: "693a9c101b7c4ee9b99d9df31d5c2bb7" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 41103 } } peers { permanent_uuid: "010b4a8232d24694a74144ef8be615ee" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38955 } }
I20260807 06:26:46.162735 10424 leader_election.cc:288] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 693a9c101b7c4ee9b99d9df31d5c2bb7 (127.15.254.1:41103), 010b4a8232d24694a74144ef8be615ee (127.15.254.2:38955)
I20260807 06:26:46.167814 10406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:46.163104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:49955 (local address 127.15.254.1:41103)
0807 06:26:46.163230 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:46.163235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:46.163249 (+ 14us) server_negotiation.cc:407] Connection header received
0807 06:26:46.163302 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:46.163306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:46.163353 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:46.163417 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:46.163758 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.164583 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.166469 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.167111 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.167229 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:46.167281 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:46.167609 (+ 328us) server_negotiation.cc:299] Negotiation successful
0807 06:26:46.167684 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 06:26:46.167814 10409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:46.163159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:52879 (local address 127.15.254.2:38955)
0807 06:26:46.163276 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:46.163281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:46.163502 (+ 221us) server_negotiation.cc:407] Connection header received
0807 06:26:46.163576 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:46.163578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:46.163617 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:46.163675 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:46.163929 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.164682 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.166466 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.167103 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.167228 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:46.167281 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:46.167609 (+ 328us) server_negotiation.cc:299] Negotiation successful
0807 06:26:46.167682 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 06:26:46.167834 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:46.162932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:41103 (local address 127.15.254.3:49955)
0807 06:26:46.163078 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:46.163096 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:46.163185 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:46.163463 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:46.163468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:46.163487 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:46.163627 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:46.163633 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.164721 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.164724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:46.166319 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:46.166339 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.167256 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.167261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:46.167454 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:46.167498 (+ 44us) client_negotiation.cc:769] Sending connection context
0807 06:26:46.167589 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 06:26:46.167685 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":209}
I20260807 06:26:46.167915 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:46.162965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:38955 (local address 127.15.254.3:52879)
0807 06:26:46.163377 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:46.163387 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:46.163503 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:46.163706 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:46.163711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:46.163731 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:46.163830 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:46.163835 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.164800 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.164803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:46.166319 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:46.166332 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.167256 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.167261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:46.167459 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:46.167502 (+ 43us) client_negotiation.cc:769] Sending connection context
0807 06:26:46.167592 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 06:26:46.167685 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":123,"spinlock_wait_cycles":768,"thread_start_us":133,"threads_started":1}
I20260807 06:26:46.168598 10148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0488b855abb4f0da1bd01e2a7b2c851" candidate_uuid: "41b1ef9463764e4eaca0f44e150e0f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "010b4a8232d24694a74144ef8be615ee" is_pre_election: true
I20260807 06:26:46.168673 9953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0488b855abb4f0da1bd01e2a7b2c851" candidate_uuid: "41b1ef9463764e4eaca0f44e150e0f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "693a9c101b7c4ee9b99d9df31d5c2bb7" is_pre_election: true
I20260807 06:26:46.168910 10148 raft_consensus.cc:2454] T d0488b855abb4f0da1bd01e2a7b2c851 P 010b4a8232d24694a74144ef8be615ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41b1ef9463764e4eaca0f44e150e0f3f in term 0.
I20260807 06:26:46.168951 9953 raft_consensus.cc:2454] T d0488b855abb4f0da1bd01e2a7b2c851 P 693a9c101b7c4ee9b99d9df31d5c2bb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41b1ef9463764e4eaca0f44e150e0f3f in term 0.
I20260807 06:26:46.169426 10238 leader_election.cc:302] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 010b4a8232d24694a74144ef8be615ee, 41b1ef9463764e4eaca0f44e150e0f3f; no voters:
I20260807 06:26:46.169709 10424 raft_consensus.cc:2781] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:26:46.169813 10424 raft_consensus.cc:482] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:26:46.169864 10424 raft_consensus.cc:3037] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:46.170917 10424 raft_consensus.cc:504] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1ef9463764e4eaca0f44e150e0f3f" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 41287 } } peers { permanent_uuid: "693a9c101b7c4ee9b99d9df31d5c2bb7" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 41103 } } peers { permanent_uuid: "010b4a8232d24694a74144ef8be615ee" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38955 } }
I20260807 06:26:46.171262 10424 leader_election.cc:288] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f [CANDIDATE]: Term 1 election: Requested vote from peers 693a9c101b7c4ee9b99d9df31d5c2bb7 (127.15.254.1:41103), 010b4a8232d24694a74144ef8be615ee (127.15.254.2:38955)
I20260807 06:26:46.171635 9953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0488b855abb4f0da1bd01e2a7b2c851" candidate_uuid: "41b1ef9463764e4eaca0f44e150e0f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "693a9c101b7c4ee9b99d9df31d5c2bb7"
I20260807 06:26:46.171687 10148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0488b855abb4f0da1bd01e2a7b2c851" candidate_uuid: "41b1ef9463764e4eaca0f44e150e0f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "010b4a8232d24694a74144ef8be615ee"
I20260807 06:26:46.171767 9953 raft_consensus.cc:3037] T d0488b855abb4f0da1bd01e2a7b2c851 P 693a9c101b7c4ee9b99d9df31d5c2bb7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:46.171800 10148 raft_consensus.cc:3037] T d0488b855abb4f0da1bd01e2a7b2c851 P 010b4a8232d24694a74144ef8be615ee [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:46.173039 9953 raft_consensus.cc:2454] T d0488b855abb4f0da1bd01e2a7b2c851 P 693a9c101b7c4ee9b99d9df31d5c2bb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41b1ef9463764e4eaca0f44e150e0f3f in term 1.
I20260807 06:26:46.173054 10148 raft_consensus.cc:2454] T d0488b855abb4f0da1bd01e2a7b2c851 P 010b4a8232d24694a74144ef8be615ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41b1ef9463764e4eaca0f44e150e0f3f in term 1.
I20260807 06:26:46.173426 10239 leader_election.cc:302] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41b1ef9463764e4eaca0f44e150e0f3f, 693a9c101b7c4ee9b99d9df31d5c2bb7; no voters:
I20260807 06:26:46.173622 10424 raft_consensus.cc:2781] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:26:46.173862 10424 raft_consensus.cc:686] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f [term 1 LEADER]: Becoming Leader. State: Replica: 41b1ef9463764e4eaca0f44e150e0f3f, State: Running, Role: LEADER
I20260807 06:26:46.174222 10424 consensus_queue.cc:237] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1ef9463764e4eaca0f44e150e0f3f" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 41287 } } peers { permanent_uuid: "693a9c101b7c4ee9b99d9df31d5c2bb7" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 41103 } } peers { permanent_uuid: "010b4a8232d24694a74144ef8be615ee" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38955 } }
I20260807 06:26:46.177536 9754 catalog_manager.cc:5495] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 41b1ef9463764e4eaca0f44e150e0f3f (127.15.254.3). New cstate: current_term: 1 leader_uuid: "41b1ef9463764e4eaca0f44e150e0f3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b1ef9463764e4eaca0f44e150e0f3f" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 41287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "693a9c101b7c4ee9b99d9df31d5c2bb7" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 41103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "010b4a8232d24694a74144ef8be615ee" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38955 } health_report { overall_health: UNKNOWN } } }
I20260807 06:26:46.199581 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:46.195533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57548 (local address 127.15.254.3:41287)
0807 06:26:46.195660 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:46.195663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:46.195678 (+ 15us) server_negotiation.cc:407] Connection header received
0807 06:26:46.195753 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:46.195755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:46.195809 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:46.195964 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:26:46.196212 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.196811 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.197818 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.198092 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.198237 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:46.198590 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:26:46.199083 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:26:46.199202 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:46.199372 (+ 170us) server_negotiation.cc:299] Negotiation successful
0807 06:26:46.199413 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 06:26:46.206511 10148 raft_consensus.cc:1261] T d0488b855abb4f0da1bd01e2a7b2c851 P 010b4a8232d24694a74144ef8be615ee [term 1 FOLLOWER]: Refusing update from remote peer 41b1ef9463764e4eaca0f44e150e0f3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:26:46.206511 9953 raft_consensus.cc:1261] T d0488b855abb4f0da1bd01e2a7b2c851 P 693a9c101b7c4ee9b99d9df31d5c2bb7 [term 1 FOLLOWER]: Refusing update from remote peer 41b1ef9463764e4eaca0f44e150e0f3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:26:46.207340 10426 consensus_queue.cc:1035] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "693a9c101b7c4ee9b99d9df31d5c2bb7" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 41103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:26:46.207580 10424 consensus_queue.cc:1035] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "010b4a8232d24694a74144ef8be615ee" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 38955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:26:46.214519 10431 mvcc.cc:205] Tried to move back new op lower bound from 7315800089413832704 to 7315800089292795904. Current Snapshot: MvccSnapshot[applied={T|T < 7315800089413832704}]
I20260807 06:26:46.217535 10432 mvcc.cc:205] Tried to move back new op lower bound from 7315800089413832704 to 7315800089292795904. Current Snapshot: MvccSnapshot[applied={T|T < 7315800089413832704}]
I20260807 06:26:46.224614 9754 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:26:46.224803 9754 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:26:46.227593 9754 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bead17ee879344dcb976054785208c94: Sending DeleteTablet for 3 replicas of tablet d0488b855abb4f0da1bd01e2a7b2c851
I20260807 06:26:46.228363 10119 tablet_service.cc:1514] Processing DeleteTablet for tablet d0488b855abb4f0da1bd01e2a7b2c851 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:26:46 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:56508
I20260807 06:26:46.228367 10304 tablet_service.cc:1514] Processing DeleteTablet for tablet d0488b855abb4f0da1bd01e2a7b2c851 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:26:46 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:57534
I20260807 06:26:46.228359 9926 tablet_service.cc:1514] Processing DeleteTablet for tablet d0488b855abb4f0da1bd01e2a7b2c851 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:26:46 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:41682
I20260807 06:26:46.228981 10449 tablet_replica.cc:330] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f: stopping tablet replica
I20260807 06:26:46.229090 10451 tablet_replica.cc:330] T d0488b855abb4f0da1bd01e2a7b2c851 P 693a9c101b7c4ee9b99d9df31d5c2bb7: stopping tablet replica
I20260807 06:26:46.229396 10449 raft_consensus.cc:2229] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:26:46.229758 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 9844
I20260807 06:26:46.229823 10449 raft_consensus.cc:2258] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:26:46.232081 10449 ts_tablet_manager.cc:1905] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:26:46.236286 10449 ts_tablet_manager.cc:1918] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:26:46.236410 10449 log.cc:1196] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/d0488b855abb4f0da1bd01e2a7b2c851
I20260807 06:26:46.236806 10449 ts_tablet_manager.cc:1939] T d0488b855abb4f0da1bd01e2a7b2c851 P 41b1ef9463764e4eaca0f44e150e0f3f: Deleting consensus metadata
I20260807 06:26:46.238233 9737 catalog_manager.cc:4841] TS 41b1ef9463764e4eaca0f44e150e0f3f (127.15.254.3:41287): tablet d0488b855abb4f0da1bd01e2a7b2c851 (table test-table [id=7c93e2db74d046ca8b5167e926efe484]) successfully deleted
I20260807 06:26:46.240032 10453 tablet_replica.cc:330] T d0488b855abb4f0da1bd01e2a7b2c851 P 010b4a8232d24694a74144ef8be615ee: stopping tablet replica
I20260807 06:26:46.240373 10453 raft_consensus.cc:2229] T d0488b855abb4f0da1bd01e2a7b2c851 P 010b4a8232d24694a74144ef8be615ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:26:46.240757 10453 raft_consensus.cc:2258] T d0488b855abb4f0da1bd01e2a7b2c851 P 010b4a8232d24694a74144ef8be615ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:26:46.242461 10453 ts_tablet_manager.cc:1905] T d0488b855abb4f0da1bd01e2a7b2c851 P 010b4a8232d24694a74144ef8be615ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:26:46.246090 10453 ts_tablet_manager.cc:1918] T d0488b855abb4f0da1bd01e2a7b2c851 P 010b4a8232d24694a74144ef8be615ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:26:46.246223 10453 log.cc:1196] T d0488b855abb4f0da1bd01e2a7b2c851 P 010b4a8232d24694a74144ef8be615ee: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/d0488b855abb4f0da1bd01e2a7b2c851
I20260807 06:26:46.246606 10453 ts_tablet_manager.cc:1939] T d0488b855abb4f0da1bd01e2a7b2c851 P 010b4a8232d24694a74144ef8be615ee: Deleting consensus metadata
I20260807 06:26:46.248018 9739 catalog_manager.cc:4841] TS 010b4a8232d24694a74144ef8be615ee (127.15.254.2:38955): tablet d0488b855abb4f0da1bd01e2a7b2c851 (table test-table [id=7c93e2db74d046ca8b5167e926efe484]) successfully deleted
W20260807 06:26:46.249212 9740 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.254.1:41103 (error 108)
I20260807 06:26:46.249394 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 10022
I20260807 06:26:46.255127 10404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:46.254826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:41103 (local address 127.0.0.1:41684)
0807 06:26:46.254969 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:46.255047 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.1:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 06:26:46.255335 9740 catalog_manager.cc:4584] TS 693a9c101b7c4ee9b99d9df31d5c2bb7 (127.15.254.1:41103): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d0488b855abb4f0da1bd01e2a7b2c851: Network error: Client connection negotiation failed: client connection to 127.15.254.1:41103: connect: Connection refused (error 111)
I20260807 06:26:46.263753 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 10215
I20260807 06:26:46.275411 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 9716
2026-08-07T06:26:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2568 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-07T06:26:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:26:46Z Disabled control of system clock
I20260807 06:26:46.314563 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:43641
--webserver_interface=127.15.254.62
--webserver_port=0
--builtin_ntp_servers=127.15.254.20:45601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:43641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 06:26:46.433604 10459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:46.433900 10459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:46.433940 10459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:46.438277 10459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:26:46.438338 10459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:46.438354 10459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:46.438370 10459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:26:46.438386 10459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:26:46.443228 10459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:45601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:43641
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:43641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:46.444307 10459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:46.445672 10459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:46.454649 10469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:46.454656 10467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:46.454710 10466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:46.454810 10459 server_base.cc:1034] running on GCE node
I20260807 06:26:46.455634 10459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:46.456876 10459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:46.458069 10459 hybrid_clock.cc:648] HybridClock initialized: now 1786084006458029 us; error 43 us; skew 500 ppm
I20260807 06:26:46.460085 10459 webserver.cc:460] Webserver started at http://127.15.254.62:41789/ using document root <none> and password file <none>
I20260807 06:26:46.460752 10459 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:46.460817 10459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:46.461082 10459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:46.463186 10459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "0f095145025f48419f59472c8b8a1229"
format_stamp: "Formatted at 2026-08-07 06:26:46 on dist-test-slave-44d2"
server_key: "6c4ff008cfe105ca16d90c5a41faf852"
server_key_iv: "caa65853ed66810b55abf01577217430"
server_key_version: "encryptionkey@0"
I20260807 06:26:46.463780 10459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "0f095145025f48419f59472c8b8a1229"
format_stamp: "Formatted at 2026-08-07 06:26:46 on dist-test-slave-44d2"
server_key: "6c4ff008cfe105ca16d90c5a41faf852"
server_key_iv: "caa65853ed66810b55abf01577217430"
server_key_version: "encryptionkey@0"
I20260807 06:26:46.467626 10459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 06:26:46.470266 10480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:46.471663 10459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 06:26:46.471819 10459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "0f095145025f48419f59472c8b8a1229"
format_stamp: "Formatted at 2026-08-07 06:26:46 on dist-test-slave-44d2"
server_key: "6c4ff008cfe105ca16d90c5a41faf852"
server_key_iv: "caa65853ed66810b55abf01577217430"
server_key_version: "encryptionkey@0"
I20260807 06:26:46.471931 10459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:46.482757 10459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:46.483443 10459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:46.483642 10459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:46.491026 10459 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:43641
I20260807 06:26:46.491041 10547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:43641 every 8 connection(s)
I20260807 06:26:46.492043 10459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:26:46.495260 10548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:46.501026 10548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f095145025f48419f59472c8b8a1229: Bootstrap starting.
I20260807 06:26:46.501582 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 10459
I20260807 06:26:46.501742 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:26:46.502086 16376 external_mini_cluster.cc:1428] Setting key 4665da22e5cb2fe03cf326706bd0d278
I20260807 06:26:46.504053 10548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f095145025f48419f59472c8b8a1229: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:46.505049 10548 log.cc:824] T 00000000000000000000000000000000 P 0f095145025f48419f59472c8b8a1229: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:46.507414 10548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f095145025f48419f59472c8b8a1229: No bootstrap required, opened a new log
I20260807 06:26:46.512058 10548 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f095145025f48419f59472c8b8a1229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f095145025f48419f59472c8b8a1229" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43641 } }
I20260807 06:26:46.512338 10548 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f095145025f48419f59472c8b8a1229 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:46.512396 10548 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f095145025f48419f59472c8b8a1229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f095145025f48419f59472c8b8a1229, State: Initialized, Role: FOLLOWER
I20260807 06:26:46.512943 10548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f095145025f48419f59472c8b8a1229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f095145025f48419f59472c8b8a1229" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43641 } }
I20260807 06:26:46.513108 10548 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f095145025f48419f59472c8b8a1229 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:26:46.513175 10548 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f095145025f48419f59472c8b8a1229 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:26:46.513192 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:46.503537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56490 (local address 127.15.254.62:43641)
0807 06:26:46.504343 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:46.504366 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:46.504413 (+ 47us) server_negotiation.cc:407] Connection header received
0807 06:26:46.505305 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:46.505330 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:46.505751 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:46.506131 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:46.506579 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.508044 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.509082 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.509797 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.509980 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:46.510908 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:46.510918 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:46.510941 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:26:46.511501 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:46.511574 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:26:46.511601 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:46.511700 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:46.511852 (+ 152us) server_negotiation.cc:299] Negotiation successful
0807 06:26:46.512145 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":143,"threads_started":1}
I20260807 06:26:46.513284 10548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f095145025f48419f59472c8b8a1229 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:46.514240 10548 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f095145025f48419f59472c8b8a1229 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f095145025f48419f59472c8b8a1229" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43641 } }
I20260807 06:26:46.514595 10548 leader_election.cc:302] T 00000000000000000000000000000000 P 0f095145025f48419f59472c8b8a1229 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f095145025f48419f59472c8b8a1229; no voters:
I20260807 06:26:46.514902 10548 leader_election.cc:288] T 00000000000000000000000000000000 P 0f095145025f48419f59472c8b8a1229 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:26:46.515065 10556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f095145025f48419f59472c8b8a1229 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:26:46.515662 10556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f095145025f48419f59472c8b8a1229 [term 1 LEADER]: Becoming Leader. State: Replica: 0f095145025f48419f59472c8b8a1229, State: Running, Role: LEADER
I20260807 06:26:46.515990 10556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f095145025f48419f59472c8b8a1229 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f095145025f48419f59472c8b8a1229" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43641 } }
I20260807 06:26:46.516306 10548 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f095145025f48419f59472c8b8a1229 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:26:46.517617 10558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f095145025f48419f59472c8b8a1229 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f095145025f48419f59472c8b8a1229. Latest consensus state: current_term: 1 leader_uuid: "0f095145025f48419f59472c8b8a1229" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f095145025f48419f59472c8b8a1229" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43641 } } }
I20260807 06:26:46.517745 10558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f095145025f48419f59472c8b8a1229 [sys.catalog]: This master's current role is: LEADER
I20260807 06:26:46.518007 10557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f095145025f48419f59472c8b8a1229 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f095145025f48419f59472c8b8a1229" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f095145025f48419f59472c8b8a1229" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43641 } } }
I20260807 06:26:46.518101 10557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f095145025f48419f59472c8b8a1229 [sys.catalog]: This master's current role is: LEADER
I20260807 06:26:46.518224 10565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:26:46.521286 10565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:26:46.526948 10565 catalog_manager.cc:1347] Generated new cluster ID: cdd97937532b4eef8d188908668a724e
I20260807 06:26:46.527043 10565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:26:46.540304 10565 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:26:46.541342 10565 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:26:46.546913 10565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f095145025f48419f59472c8b8a1229: Generated new TSK 0
I20260807 06:26:46.547613 10565 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:26:46.558974 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:0
--local_ip_for_outbound_sockets=127.15.254.1
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:43641
--builtin_ntp_servers=127.15.254.20:45601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 06:26:46.691953 10577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:46.692225 10577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:46.692265 10577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:46.696296 10577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:46.696358 10577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:46.696411 10577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:26:46.701411 10577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:45601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:43641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:46.702591 10577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:46.704030 10577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:46.714520 10584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:46.714520 10585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:46.714514 10587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:46.714656 10577 server_base.cc:1034] running on GCE node
I20260807 06:26:46.715299 10577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:46.716022 10577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:46.717229 10577 hybrid_clock.cc:648] HybridClock initialized: now 1786084006717205 us; error 36 us; skew 500 ppm
I20260807 06:26:46.719492 10577 webserver.cc:460] Webserver started at http://127.15.254.1:41177/ using document root <none> and password file <none>
I20260807 06:26:46.720124 10577 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:46.720221 10577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:46.720471 10577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:46.722735 10577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-0/data/instance:
uuid: "8fd329ad1c78481395902376ffa76564"
format_stamp: "Formatted at 2026-08-07 06:26:46 on dist-test-slave-44d2"
server_key: "cd2e2f9fdeb89c017a39433a4681c0b6"
server_key_iv: "befaeeff9e634441517fed4357b18467"
server_key_version: "encryptionkey@0"
I20260807 06:26:46.723366 10577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance:
uuid: "8fd329ad1c78481395902376ffa76564"
format_stamp: "Formatted at 2026-08-07 06:26:46 on dist-test-slave-44d2"
server_key: "cd2e2f9fdeb89c017a39433a4681c0b6"
server_key_iv: "befaeeff9e634441517fed4357b18467"
server_key_version: "encryptionkey@0"
I20260807 06:26:46.727505 10577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 06:26:46.730335 10595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:46.731781 10577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:26:46.731936 10577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "8fd329ad1c78481395902376ffa76564"
format_stamp: "Formatted at 2026-08-07 06:26:46 on dist-test-slave-44d2"
server_key: "cd2e2f9fdeb89c017a39433a4681c0b6"
server_key_iv: "befaeeff9e634441517fed4357b18467"
server_key_version: "encryptionkey@0"
I20260807 06:26:46.732061 10577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:46.757038 10577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:46.757975 10577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:46.758234 10577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:46.758975 10577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:26:46.760143 10577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:26:46.760229 10577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:46.760316 10577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:26:46.760370 10577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:46.771461 10577 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:43821
I20260807 06:26:46.771487 10755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:43821 every 8 connection(s)
I20260807 06:26:46.772576 10577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:26:46.777983 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 10577
I20260807 06:26:46.778270 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance
I20260807 06:26:46.778642 16376 external_mini_cluster.cc:1428] Setting key e70405b5f492b62b501369106cabea9c
I20260807 06:26:46.780957 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:0
--local_ip_for_outbound_sockets=127.15.254.2
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:43641
--builtin_ntp_servers=127.15.254.20:45601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 06:26:46.782779 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:46.774606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:37587 (local address 127.15.254.62:43641)
0807 06:26:46.774756 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:46.774760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:46.776065 (+ 1305us) server_negotiation.cc:407] Connection header received
0807 06:26:46.777702 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:46.777705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:46.777769 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:46.777871 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:46.778936 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.779496 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.780730 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.780972 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.781083 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:46.782271 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:46.782276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:46.782278 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:26:46.782292 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:46.782319 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:26:46.782327 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:46.782404 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:46.782572 (+ 168us) server_negotiation.cc:299] Negotiation successful
0807 06:26:46.782636 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 06:26:46.783653 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:46.775086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:43641 (local address 127.15.254.1:37587)
0807 06:26:46.775820 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:46.775862 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:46.777455 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:46.778076 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:46.778087 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:46.778193 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:26:46.778791 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:46.778807 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.779633 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.779636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:46.780612 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:46.780622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.781129 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.781133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:46.781332 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:46.781982 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:26:46.781984 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:26:46.782026 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:26:46.782112 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:26:46.782406 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:26:46.782419 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:26:46.782428 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 06:26:46.782570 (+ 142us) client_negotiation.cc:240] Negotiation successful
0807 06:26:46.782869 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":137,"threads_started":1}
I20260807 06:26:46.785187 10756 heartbeater.cc:347] Connected to a master server at 127.15.254.62:43641
I20260807 06:26:46.785465 10756 heartbeater.cc:464] Registering TS with master...
I20260807 06:26:46.786103 10756 heartbeater.cc:511] Master 127.15.254.62:43641 requested a full tablet report, sending...
I20260807 06:26:46.787722 10499 ts_manager.cc:194] Registered new tserver with Master: 8fd329ad1c78481395902376ffa76564 (127.15.254.1:43821)
I20260807 06:26:46.788966 10499 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.254.1:37587
W20260807 06:26:46.901780 10759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:46.902088 10759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:46.902159 10759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:46.906186 10759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:46.906273 10759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:46.906369 10759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:26:46.911499 10759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:45601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:43641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:46.912724 10759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:46.914113 10759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:46.924597 10769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:46.924608 10767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:46.924666 10759 server_base.cc:1034] running on GCE node
W20260807 06:26:46.924603 10766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:46.925283 10759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:46.926010 10759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:46.927209 10759 hybrid_clock.cc:648] HybridClock initialized: now 1786084006927193 us; error 28 us; skew 500 ppm
I20260807 06:26:46.929352 10759 webserver.cc:460] Webserver started at http://127.15.254.2:32997/ using document root <none> and password file <none>
I20260807 06:26:46.930282 10759 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:46.930402 10759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:46.930662 10759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:46.932898 10759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-1/data/instance:
uuid: "e07905770bda4a068f7ffdae24cbd65d"
format_stamp: "Formatted at 2026-08-07 06:26:46 on dist-test-slave-44d2"
server_key: "6d07b63ada7d15a4a7fba2754b469427"
server_key_iv: "b4f0bf962d692b75ddb85fa69ddcf375"
server_key_version: "encryptionkey@0"
I20260807 06:26:46.933485 10759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance:
uuid: "e07905770bda4a068f7ffdae24cbd65d"
format_stamp: "Formatted at 2026-08-07 06:26:46 on dist-test-slave-44d2"
server_key: "6d07b63ada7d15a4a7fba2754b469427"
server_key_iv: "b4f0bf962d692b75ddb85fa69ddcf375"
server_key_version: "encryptionkey@0"
I20260807 06:26:46.937505 10759 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 06:26:46.939952 10777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:46.941336 10759 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 06:26:46.941457 10759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "e07905770bda4a068f7ffdae24cbd65d"
format_stamp: "Formatted at 2026-08-07 06:26:46 on dist-test-slave-44d2"
server_key: "6d07b63ada7d15a4a7fba2754b469427"
server_key_iv: "b4f0bf962d692b75ddb85fa69ddcf375"
server_key_version: "encryptionkey@0"
I20260807 06:26:46.941565 10759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:46.971885 10759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:46.972816 10759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:46.973052 10759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:46.973755 10759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:26:46.974869 10759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:26:46.974926 10759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:46.974997 10759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:26:46.975049 10759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:46.985517 10759 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:45383
I20260807 06:26:46.985555 10934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:45383 every 8 connection(s)
I20260807 06:26:46.986737 10759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:26:46.990980 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 10759
I20260807 06:26:46.991140 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance
I20260807 06:26:46.991462 16376 external_mini_cluster.cc:1428] Setting key 472d9c10f0573f8e8dd1885f616cbe0d
I20260807 06:26:46.993755 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:0
--local_ip_for_outbound_sockets=127.15.254.3
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:43641
--builtin_ntp_servers=127.15.254.20:45601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 06:26:46.998150 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:46.988847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:39603 (local address 127.15.254.62:43641)
0807 06:26:46.989017 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:46.989023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:46.990293 (+ 1270us) server_negotiation.cc:407] Connection header received
0807 06:26:46.991948 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:46.991951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:46.992022 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:46.992110 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:46.993327 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.994131 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.995489 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.995810 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.995898 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:46.997551 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:46.997556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:46.997564 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:26:46.997581 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:46.997605 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:26:46.997614 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:46.997714 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:46.997901 (+ 187us) server_negotiation.cc:299] Negotiation successful
0807 06:26:46.997978 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 06:26:46.999116 10938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:46.989333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:43641 (local address 127.15.254.2:39603)
0807 06:26:46.990132 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:46.990177 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:46.991707 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:46.992366 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:46.992374 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:46.992522 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:26:46.993181 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:46.993197 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.994293 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.994296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:46.995339 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:46.995352 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:46.996055 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:46.996061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:46.996339 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:46.997163 (+ 824us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:26:46.997166 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:26:46.997231 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:26:46.997351 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:26:46.997716 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:26:46.997730 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:26:46.997741 (+ 11us) client_negotiation.cc:769] Sending connection context
0807 06:26:46.997899 (+ 158us) client_negotiation.cc:240] Negotiation successful
0807 06:26:46.998227 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":184,"threads_started":1}
I20260807 06:26:47.000622 10935 heartbeater.cc:347] Connected to a master server at 127.15.254.62:43641
I20260807 06:26:47.001011 10935 heartbeater.cc:464] Registering TS with master...
I20260807 06:26:47.001685 10935 heartbeater.cc:511] Master 127.15.254.62:43641 requested a full tablet report, sending...
I20260807 06:26:47.003094 10499 ts_manager.cc:194] Registered new tserver with Master: e07905770bda4a068f7ffdae24cbd65d (127.15.254.2:45383)
I20260807 06:26:47.004040 10499 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.254.2:39603
W20260807 06:26:47.114315 10939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:47.114583 10939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:47.114620 10939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:47.118729 10939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:47.118791 10939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:47.118846 10939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:26:47.123893 10939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:45601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:43641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:47.125095 10939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:47.126422 10939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:47.136732 10949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:47.136895 10939 server_base.cc:1034] running on GCE node
W20260807 06:26:47.136739 10948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:47.137092 10952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:47.137573 10939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:47.138214 10939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:47.139379 10939 hybrid_clock.cc:648] HybridClock initialized: now 1786084007139352 us; error 36 us; skew 500 ppm
I20260807 06:26:47.141418 10939 webserver.cc:460] Webserver started at http://127.15.254.3:43875/ using document root <none> and password file <none>
I20260807 06:26:47.142058 10939 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:47.142153 10939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:47.142395 10939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:47.144590 10939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-2/data/instance:
uuid: "3b743124e345452c909740ec9a34fd22"
format_stamp: "Formatted at 2026-08-07 06:26:47 on dist-test-slave-44d2"
server_key: "e61a4619731d14bfdb972776aaf90482"
server_key_iv: "13076a210276bca7be86d738510be45b"
server_key_version: "encryptionkey@0"
I20260807 06:26:47.145226 10939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance:
uuid: "3b743124e345452c909740ec9a34fd22"
format_stamp: "Formatted at 2026-08-07 06:26:47 on dist-test-slave-44d2"
server_key: "e61a4619731d14bfdb972776aaf90482"
server_key_iv: "13076a210276bca7be86d738510be45b"
server_key_version: "encryptionkey@0"
I20260807 06:26:47.149137 10939 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 06:26:47.151666 10958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:47.153072 10939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 06:26:47.153230 10939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "3b743124e345452c909740ec9a34fd22"
format_stamp: "Formatted at 2026-08-07 06:26:47 on dist-test-slave-44d2"
server_key: "e61a4619731d14bfdb972776aaf90482"
server_key_iv: "13076a210276bca7be86d738510be45b"
server_key_version: "encryptionkey@0"
I20260807 06:26:47.153357 10939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:47.170158 10939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:47.170888 10939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:47.171092 10939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:47.171710 10939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:26:47.172722 10939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:26:47.172792 10939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:47.172863 10939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:26:47.172909 10939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:47.182317 10939 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:36049
I20260807 06:26:47.182387 11108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:36049 every 8 connection(s)
I20260807 06:26:47.183324 10939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:26:47.192404 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 10939
I20260807 06:26:47.192580 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance
I20260807 06:26:47.192943 16376 external_mini_cluster.cc:1428] Setting key cc306c3359373e95f1bd0d5c80d32ea8
I20260807 06:26:47.193194 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:47.185297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:58815 (local address 127.15.254.62:43641)
0807 06:26:47.185465 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:47.185469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:47.186547 (+ 1078us) server_negotiation.cc:407] Connection header received
0807 06:26:47.188057 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:47.188060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:47.188118 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:47.188227 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:47.189277 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.189818 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.190993 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.191220 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.191328 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:47.192438 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:47.192441 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:47.192443 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:26:47.192455 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:47.192519 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:26:47.192532 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:47.192614 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:47.192888 (+ 274us) server_negotiation.cc:299] Negotiation successful
0807 06:26:47.193033 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 06:26:47.194269 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:47.185697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:43641 (local address 127.15.254.3:58815)
0807 06:26:47.186388 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:47.186433 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:47.187783 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:47.188450 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:47.188457 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:47.188599 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:26:47.189133 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:47.189146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.189956 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.189959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:47.190860 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:47.190870 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.191376 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.191380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:47.191550 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:47.192148 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:26:47.192150 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:26:47.192190 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:26:47.192255 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:26:47.192663 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:26:47.192674 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:26:47.192681 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 06:26:47.192833 (+ 152us) client_negotiation.cc:240] Negotiation successful
0807 06:26:47.193189 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":119,"threads_started":1}
I20260807 06:26:47.195465 11110 heartbeater.cc:347] Connected to a master server at 127.15.254.62:43641
I20260807 06:26:47.195674 11110 heartbeater.cc:464] Registering TS with master...
I20260807 06:26:47.196165 11110 heartbeater.cc:511] Master 127.15.254.62:43641 requested a full tablet report, sending...
I20260807 06:26:47.197268 10498 ts_manager.cc:194] Registered new tserver with Master: 3b743124e345452c909740ec9a34fd22 (127.15.254.3:36049)
I20260807 06:26:47.197861 10498 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.254.3:58815
I20260807 06:26:47.207242 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:26:47.213717 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:47.209365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56494 (local address 127.15.254.62:43641)
0807 06:26:47.209524 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:47.209529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:47.209733 (+ 204us) server_negotiation.cc:407] Connection header received
0807 06:26:47.209926 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:47.209928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:47.209980 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:47.210058 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:47.210351 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.211032 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.211947 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.212187 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.212281 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:47.213232 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:47.213235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:47.213238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:26:47.213250 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:47.213270 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:26:47.213278 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:47.213390 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:47.213493 (+ 103us) server_negotiation.cc:299] Negotiation successful
0807 06:26:47.213554 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 06:26:47.217254 10498 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:26:47.220520 10498 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:26:47.235610 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:47.230050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:36049 (local address 127.0.0.1:42648)
0807 06:26:47.230386 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:47.230401 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:47.230502 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:47.231188 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:47.231191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:47.231218 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:47.231379 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:47.231386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.232615 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.232618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:47.234003 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:47.234013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.235002 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.235006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:47.235141 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:47.235196 (+ 55us) client_negotiation.cc:769] Sending connection context
0807 06:26:47.235311 (+ 115us) client_negotiation.cc:240] Negotiation successful
0807 06:26:47.235442 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":118,"spinlock_wait_cycles":768,"thread_start_us":99,"threads_started":1}
I20260807 06:26:47.235886 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:47.230283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42648 (local address 127.15.254.3:36049)
0807 06:26:47.230778 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:47.230790 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:47.230811 (+ 21us) server_negotiation.cc:407] Connection header received
0807 06:26:47.230880 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:47.230884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:47.231038 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:47.231531 (+ 493us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:47.231665 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.232401 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.234125 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.234862 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.234976 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:47.235185 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:47.235360 (+ 175us) server_negotiation.cc:299] Negotiation successful
0807 06:26:47.235480 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":177,"threads_started":1}
I20260807 06:26:47.236235 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:47.229426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56228 (local address 127.15.254.2:45383)
0807 06:26:47.230194 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:47.230207 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:47.230234 (+ 27us) server_negotiation.cc:407] Connection header received
0807 06:26:47.230801 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:47.230807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:47.231007 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:47.231143 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:47.231608 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.232811 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.234448 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.235235 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.235362 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:47.235536 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:47.235906 (+ 370us) server_negotiation.cc:299] Negotiation successful
0807 06:26:47.236030 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":110,"threads_started":1}
I20260807 06:26:47.236304 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:47.229556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48998 (local address 127.15.254.1:43821)
0807 06:26:47.229989 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:47.230002 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:47.230635 (+ 633us) server_negotiation.cc:407] Connection header received
0807 06:26:47.230724 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:47.230730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:47.230974 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:47.231151 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:47.231504 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.232811 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.234483 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.235074 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.235178 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:47.235364 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:47.236013 (+ 649us) server_negotiation.cc:299] Negotiation successful
0807 06:26:47.236126 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":100,"threads_started":1}
I20260807 06:26:47.236709 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:47.229261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:45383 (local address 127.0.0.1:56228)
0807 06:26:47.229948 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:47.230002 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:47.230144 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:47.231334 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:47.231337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:47.231352 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:47.231437 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:47.231443 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.232934 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.232937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:47.234310 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:47.234322 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.235605 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.235611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:47.235714 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:47.235748 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 06:26:47.236427 (+ 679us) client_negotiation.cc:240] Negotiation successful
0807 06:26:47.236561 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":52,"thread_start_us":94,"threads_started":1}
I20260807 06:26:47.237044 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:47.229261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:43821 (local address 127.0.0.1:48998)
0807 06:26:47.230356 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:47.230371 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:47.230474 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:47.231144 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:47.231150 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:47.231173 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:47.231378 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:47.231386 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.233095 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.233098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:47.234328 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:47.234335 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.235426 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.235429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:47.235577 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:47.235604 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 06:26:47.235677 (+ 73us) client_negotiation.cc:240] Negotiation successful
0807 06:26:47.235949 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"mutex_wait_us":193,"spinlock_wait_cycles":4992,"thread_start_us":119,"threads_started":1}
I20260807 06:26:47.239424 10667 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea880b202b8442fbaa96f9317818fdd (DEFAULT_TABLE table=test-table [id=d24e426398824f9e990b5cee51552503]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:26:47.239326 11019 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea880b202b8442fbaa96f9317818fdd (DEFAULT_TABLE table=test-table [id=d24e426398824f9e990b5cee51552503]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:26:47.239606 10842 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea880b202b8442fbaa96f9317818fdd (DEFAULT_TABLE table=test-table [id=d24e426398824f9e990b5cee51552503]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:26:47.240413 10667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ea880b202b8442fbaa96f9317818fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:47.240430 11019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ea880b202b8442fbaa96f9317818fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:47.240466 10842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ea880b202b8442fbaa96f9317818fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:47.247095 11132 tablet_bootstrap.cc:492] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564: Bootstrap starting.
I20260807 06:26:47.248608 11134 tablet_bootstrap.cc:492] T 7ea880b202b8442fbaa96f9317818fdd P e07905770bda4a068f7ffdae24cbd65d: Bootstrap starting.
I20260807 06:26:47.249969 11132 tablet_bootstrap.cc:654] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:47.250819 11134 tablet_bootstrap.cc:654] T 7ea880b202b8442fbaa96f9317818fdd P e07905770bda4a068f7ffdae24cbd65d: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:47.250911 11132 log.cc:824] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:47.251711 11134 log.cc:824] T 7ea880b202b8442fbaa96f9317818fdd P e07905770bda4a068f7ffdae24cbd65d: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:47.252265 11133 tablet_bootstrap.cc:492] T 7ea880b202b8442fbaa96f9317818fdd P 3b743124e345452c909740ec9a34fd22: Bootstrap starting.
I20260807 06:26:47.253222 11132 tablet_bootstrap.cc:492] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564: No bootstrap required, opened a new log
I20260807 06:26:47.253443 11132 ts_tablet_manager.cc:1397] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260807 06:26:47.253907 11134 tablet_bootstrap.cc:492] T 7ea880b202b8442fbaa96f9317818fdd P e07905770bda4a068f7ffdae24cbd65d: No bootstrap required, opened a new log
I20260807 06:26:47.254103 11134 ts_tablet_manager.cc:1397] T 7ea880b202b8442fbaa96f9317818fdd P e07905770bda4a068f7ffdae24cbd65d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 06:26:47.254952 11133 tablet_bootstrap.cc:654] T 7ea880b202b8442fbaa96f9317818fdd P 3b743124e345452c909740ec9a34fd22: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:47.255776 11133 log.cc:824] T 7ea880b202b8442fbaa96f9317818fdd P 3b743124e345452c909740ec9a34fd22: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:47.257159 11132 raft_consensus.cc:348] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07905770bda4a068f7ffdae24cbd65d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 45383 } } peers { permanent_uuid: "8fd329ad1c78481395902376ffa76564" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 43821 } } peers { permanent_uuid: "3b743124e345452c909740ec9a34fd22" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36049 } }
I20260807 06:26:47.257472 11132 raft_consensus.cc:374] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:47.257529 11132 raft_consensus.cc:729] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fd329ad1c78481395902376ffa76564, State: Initialized, Role: FOLLOWER
I20260807 06:26:47.257949 11133 tablet_bootstrap.cc:492] T 7ea880b202b8442fbaa96f9317818fdd P 3b743124e345452c909740ec9a34fd22: No bootstrap required, opened a new log
I20260807 06:26:47.258136 11133 ts_tablet_manager.cc:1397] T 7ea880b202b8442fbaa96f9317818fdd P 3b743124e345452c909740ec9a34fd22: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260807 06:26:47.258064 11132 consensus_queue.cc:260] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07905770bda4a068f7ffdae24cbd65d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 45383 } } peers { permanent_uuid: "8fd329ad1c78481395902376ffa76564" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 43821 } } peers { permanent_uuid: "3b743124e345452c909740ec9a34fd22" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36049 } }
I20260807 06:26:47.258764 11134 raft_consensus.cc:348] T 7ea880b202b8442fbaa96f9317818fdd P e07905770bda4a068f7ffdae24cbd65d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07905770bda4a068f7ffdae24cbd65d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 45383 } } peers { permanent_uuid: "8fd329ad1c78481395902376ffa76564" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 43821 } } peers { permanent_uuid: "3b743124e345452c909740ec9a34fd22" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36049 } }
I20260807 06:26:47.259032 11134 raft_consensus.cc:374] T 7ea880b202b8442fbaa96f9317818fdd P e07905770bda4a068f7ffdae24cbd65d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:47.259096 11134 raft_consensus.cc:729] T 7ea880b202b8442fbaa96f9317818fdd P e07905770bda4a068f7ffdae24cbd65d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07905770bda4a068f7ffdae24cbd65d, State: Initialized, Role: FOLLOWER
I20260807 06:26:47.259101 11132 ts_tablet_manager.cc:1428] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260807 06:26:47.259470 10756 heartbeater.cc:503] Master 127.15.254.62:43641 was elected leader, sending a full tablet report...
I20260807 06:26:47.259546 11134 consensus_queue.cc:260] T 7ea880b202b8442fbaa96f9317818fdd P e07905770bda4a068f7ffdae24cbd65d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07905770bda4a068f7ffdae24cbd65d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 45383 } } peers { permanent_uuid: "8fd329ad1c78481395902376ffa76564" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 43821 } } peers { permanent_uuid: "3b743124e345452c909740ec9a34fd22" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36049 } }
I20260807 06:26:47.260262 10935 heartbeater.cc:503] Master 127.15.254.62:43641 was elected leader, sending a full tablet report...
I20260807 06:26:47.260573 11134 ts_tablet_manager.cc:1428] T 7ea880b202b8442fbaa96f9317818fdd P e07905770bda4a068f7ffdae24cbd65d: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260807 06:26:47.262719 11133 raft_consensus.cc:348] T 7ea880b202b8442fbaa96f9317818fdd P 3b743124e345452c909740ec9a34fd22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07905770bda4a068f7ffdae24cbd65d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 45383 } } peers { permanent_uuid: "8fd329ad1c78481395902376ffa76564" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 43821 } } peers { permanent_uuid: "3b743124e345452c909740ec9a34fd22" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36049 } }
I20260807 06:26:47.262950 11133 raft_consensus.cc:374] T 7ea880b202b8442fbaa96f9317818fdd P 3b743124e345452c909740ec9a34fd22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:47.263011 11133 raft_consensus.cc:729] T 7ea880b202b8442fbaa96f9317818fdd P 3b743124e345452c909740ec9a34fd22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b743124e345452c909740ec9a34fd22, State: Initialized, Role: FOLLOWER
I20260807 06:26:47.263481 11133 consensus_queue.cc:260] T 7ea880b202b8442fbaa96f9317818fdd P 3b743124e345452c909740ec9a34fd22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07905770bda4a068f7ffdae24cbd65d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 45383 } } peers { permanent_uuid: "8fd329ad1c78481395902376ffa76564" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 43821 } } peers { permanent_uuid: "3b743124e345452c909740ec9a34fd22" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36049 } }
I20260807 06:26:47.264448 11133 ts_tablet_manager.cc:1428] T 7ea880b202b8442fbaa96f9317818fdd P 3b743124e345452c909740ec9a34fd22: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260807 06:26:47.264787 11110 heartbeater.cc:503] Master 127.15.254.62:43641 was elected leader, sending a full tablet report...
W20260807 06:26:47.273989 10757 tablet.cc:2369] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 06:26:47.434404 11111 tablet.cc:2369] T 7ea880b202b8442fbaa96f9317818fdd P 3b743124e345452c909740ec9a34fd22: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 06:26:47.488276 10937 tablet.cc:2369] T 7ea880b202b8442fbaa96f9317818fdd P e07905770bda4a068f7ffdae24cbd65d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:26:47.499156 11138 raft_consensus.cc:482] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:26:47.499430 11138 raft_consensus.cc:504] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07905770bda4a068f7ffdae24cbd65d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 45383 } } peers { permanent_uuid: "8fd329ad1c78481395902376ffa76564" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 43821 } } peers { permanent_uuid: "3b743124e345452c909740ec9a34fd22" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36049 } }
I20260807 06:26:47.500903 11138 leader_election.cc:288] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07905770bda4a068f7ffdae24cbd65d (127.15.254.2:45383), 3b743124e345452c909740ec9a34fd22 (127.15.254.3:36049)
I20260807 06:26:47.505606 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:47.501782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:44933 (local address 127.15.254.3:36049)
0807 06:26:47.501917 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:47.501924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:47.502171 (+ 247us) server_negotiation.cc:407] Connection header received
0807 06:26:47.502314 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:47.502317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:47.502366 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:47.502454 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:47.502765 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.503299 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.504606 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.505029 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.505170 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:47.505225 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:47.505406 (+ 181us) server_negotiation.cc:299] Negotiation successful
0807 06:26:47.505483 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 06:26:47.505630 11147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:47.501672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:36049 (local address 127.15.254.1:44933)
0807 06:26:47.502075 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:47.502086 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:47.502215 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:47.502466 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:47.502469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:47.502486 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:47.502652 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:47.502659 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.503420 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.503424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:47.504425 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:47.504437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.505170 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.505173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:47.505272 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:47.505296 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 06:26:47.505388 (+ 92us) client_negotiation.cc:240] Negotiation successful
0807 06:26:47.505459 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":66,"thread_start_us":122,"threads_started":1}
I20260807 06:26:47.505904 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:47.501292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:38737 (local address 127.15.254.2:45383)
0807 06:26:47.501438 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:47.501443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:47.501711 (+ 268us) server_negotiation.cc:407] Connection header received
0807 06:26:47.501940 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:47.501944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:47.502002 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:47.502055 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:47.502551 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.503363 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.504896 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.505290 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.505453 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:47.505509 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:47.505697 (+ 188us) server_negotiation.cc:299] Negotiation successful
0807 06:26:47.505774 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 06:26:47.505928 11143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:47.501185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:45383 (local address 127.15.254.1:38737)
0807 06:26:47.501602 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:47.501619 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:47.501803 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:47.502227 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:47.502231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:47.502250 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:47.502407 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:47.502414 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.503521 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.503525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:47.504752 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:47.504764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.505453 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.505458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:47.505563 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:47.505583 (+ 20us) client_negotiation.cc:769] Sending connection context
0807 06:26:47.505680 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 06:26:47.505753 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":18,"thread_start_us":113,"threads_started":1}
I20260807 06:26:47.506424 11047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea880b202b8442fbaa96f9317818fdd" candidate_uuid: "8fd329ad1c78481395902376ffa76564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b743124e345452c909740ec9a34fd22" is_pre_election: true
I20260807 06:26:47.506541 10867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea880b202b8442fbaa96f9317818fdd" candidate_uuid: "8fd329ad1c78481395902376ffa76564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07905770bda4a068f7ffdae24cbd65d" is_pre_election: true
I20260807 06:26:47.506779 10867 raft_consensus.cc:2454] T 7ea880b202b8442fbaa96f9317818fdd P e07905770bda4a068f7ffdae24cbd65d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fd329ad1c78481395902376ffa76564 in term 0.
I20260807 06:26:47.506771 11047 raft_consensus.cc:2454] T 7ea880b202b8442fbaa96f9317818fdd P 3b743124e345452c909740ec9a34fd22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fd329ad1c78481395902376ffa76564 in term 0.
I20260807 06:26:47.507383 10596 leader_election.cc:302] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b743124e345452c909740ec9a34fd22, 8fd329ad1c78481395902376ffa76564; no voters:
I20260807 06:26:47.507694 11138 raft_consensus.cc:2781] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:26:47.507799 11138 raft_consensus.cc:482] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:26:47.507858 11138 raft_consensus.cc:3037] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:47.509186 11138 raft_consensus.cc:504] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07905770bda4a068f7ffdae24cbd65d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 45383 } } peers { permanent_uuid: "8fd329ad1c78481395902376ffa76564" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 43821 } } peers { permanent_uuid: "3b743124e345452c909740ec9a34fd22" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36049 } }
I20260807 06:26:47.509562 11138 leader_election.cc:288] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564 [CANDIDATE]: Term 1 election: Requested vote from peers e07905770bda4a068f7ffdae24cbd65d (127.15.254.2:45383), 3b743124e345452c909740ec9a34fd22 (127.15.254.3:36049)
I20260807 06:26:47.510022 10867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea880b202b8442fbaa96f9317818fdd" candidate_uuid: "8fd329ad1c78481395902376ffa76564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07905770bda4a068f7ffdae24cbd65d"
I20260807 06:26:47.510028 11047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea880b202b8442fbaa96f9317818fdd" candidate_uuid: "8fd329ad1c78481395902376ffa76564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b743124e345452c909740ec9a34fd22"
I20260807 06:26:47.510176 11047 raft_consensus.cc:3037] T 7ea880b202b8442fbaa96f9317818fdd P 3b743124e345452c909740ec9a34fd22 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:47.510167 10867 raft_consensus.cc:3037] T 7ea880b202b8442fbaa96f9317818fdd P e07905770bda4a068f7ffdae24cbd65d [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:47.511529 11047 raft_consensus.cc:2454] T 7ea880b202b8442fbaa96f9317818fdd P 3b743124e345452c909740ec9a34fd22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fd329ad1c78481395902376ffa76564 in term 1.
I20260807 06:26:47.511529 10867 raft_consensus.cc:2454] T 7ea880b202b8442fbaa96f9317818fdd P e07905770bda4a068f7ffdae24cbd65d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fd329ad1c78481395902376ffa76564 in term 1.
I20260807 06:26:47.511905 10596 leader_election.cc:302] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b743124e345452c909740ec9a34fd22, 8fd329ad1c78481395902376ffa76564; no voters:
I20260807 06:26:47.512115 11138 raft_consensus.cc:2781] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:26:47.512327 11138 raft_consensus.cc:686] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564 [term 1 LEADER]: Becoming Leader. State: Replica: 8fd329ad1c78481395902376ffa76564, State: Running, Role: LEADER
I20260807 06:26:47.512748 11138 consensus_queue.cc:237] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07905770bda4a068f7ffdae24cbd65d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 45383 } } peers { permanent_uuid: "8fd329ad1c78481395902376ffa76564" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 43821 } } peers { permanent_uuid: "3b743124e345452c909740ec9a34fd22" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36049 } }
I20260807 06:26:47.516052 10498 catalog_manager.cc:5495] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8fd329ad1c78481395902376ffa76564 (127.15.254.1). New cstate: current_term: 1 leader_uuid: "8fd329ad1c78481395902376ffa76564" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07905770bda4a068f7ffdae24cbd65d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 45383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fd329ad1c78481395902376ffa76564" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 43821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b743124e345452c909740ec9a34fd22" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36049 } health_report { overall_health: UNKNOWN } } }
I20260807 06:26:47.532069 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:47.528414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49006 (local address 127.15.254.1:43821)
0807 06:26:47.528574 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:47.528578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:47.528593 (+ 15us) server_negotiation.cc:407] Connection header received
0807 06:26:47.528731 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:47.528734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:47.528797 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:47.528890 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:26:47.529130 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.529661 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.530574 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:47.530797 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:47.530881 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:47.531203 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:26:47.531622 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:26:47.531736 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:47.531866 (+ 130us) server_negotiation.cc:299] Negotiation successful
0807 06:26:47.531952 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 06:26:47.538417 11047 raft_consensus.cc:1261] T 7ea880b202b8442fbaa96f9317818fdd P 3b743124e345452c909740ec9a34fd22 [term 1 FOLLOWER]: Refusing update from remote peer 8fd329ad1c78481395902376ffa76564: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:26:47.538417 10867 raft_consensus.cc:1261] T 7ea880b202b8442fbaa96f9317818fdd P e07905770bda4a068f7ffdae24cbd65d [term 1 FOLLOWER]: Refusing update from remote peer 8fd329ad1c78481395902376ffa76564: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:26:47.539186 11138 consensus_queue.cc:1035] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b743124e345452c909740ec9a34fd22" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 36049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:26:47.539417 11148 consensus_queue.cc:1035] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07905770bda4a068f7ffdae24cbd65d" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:26:47.547194 11156 mvcc.cc:205] Tried to move back new op lower bound from 7315800094869241856 to 7315800094775218176. Current Snapshot: MvccSnapshot[applied={T|T < 7315800094869241856}]
I20260807 06:26:47.556221 10498 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:26:47.556418 10498 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:26:47.559307 10498 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f095145025f48419f59472c8b8a1229: Sending DeleteTablet for 3 replicas of tablet 7ea880b202b8442fbaa96f9317818fdd
I20260807 06:26:47.560197 10667 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ea880b202b8442fbaa96f9317818fdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:26:47 UTC) from {username='slave'} at 127.0.0.1:48998
I20260807 06:26:47.560250 11019 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ea880b202b8442fbaa96f9317818fdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:26:47 UTC) from {username='slave'} at 127.0.0.1:42648
I20260807 06:26:47.560354 10842 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ea880b202b8442fbaa96f9317818fdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:26:47 UTC) from {username='slave'} at 127.0.0.1:56228
I20260807 06:26:47.560956 11171 tablet_replica.cc:330] T 7ea880b202b8442fbaa96f9317818fdd P 8fd329ad1c78481395902376ffa76564: stopping tablet replica
I20260807 06:26:47.560988 11170 tablet_replica.cc:330] T 7ea880b202b8442fbaa96f9317818fdd P 3b743124e345452c909740ec9a34fd22: stopping tablet replica
I20260807 06:26:47.561359 11170 raft_consensus.cc:2229] T 7ea880b202b8442fbaa96f9317818fdd P 3b743124e345452c909740ec9a34fd22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:26:47.561523 11172 tablet_replica.cc:330] T 7ea880b202b8442fbaa96f9317818fdd P e07905770bda4a068f7ffdae24cbd65d: stopping tablet replica
I20260807 06:26:47.561611 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 10577
I20260807 06:26:47.561748 11170 raft_consensus.cc:2258] T 7ea880b202b8442fbaa96f9317818fdd P 3b743124e345452c909740ec9a34fd22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:26:47.561825 11172 raft_consensus.cc:2229] T 7ea880b202b8442fbaa96f9317818fdd P e07905770bda4a068f7ffdae24cbd65d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:26:47.563994 11170 ts_tablet_manager.cc:1905] T 7ea880b202b8442fbaa96f9317818fdd P 3b743124e345452c909740ec9a34fd22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:26:47.565203 11172 raft_consensus.cc:2258] T 7ea880b202b8442fbaa96f9317818fdd P e07905770bda4a068f7ffdae24cbd65d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:26:47.566728 11172 ts_tablet_manager.cc:1905] T 7ea880b202b8442fbaa96f9317818fdd P e07905770bda4a068f7ffdae24cbd65d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:26:47.567795 11170 ts_tablet_manager.cc:1918] T 7ea880b202b8442fbaa96f9317818fdd P 3b743124e345452c909740ec9a34fd22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:26:47.567924 11170 log.cc:1196] T 7ea880b202b8442fbaa96f9317818fdd P 3b743124e345452c909740ec9a34fd22: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/7ea880b202b8442fbaa96f9317818fdd
I20260807 06:26:47.568274 11170 ts_tablet_manager.cc:1939] T 7ea880b202b8442fbaa96f9317818fdd P 3b743124e345452c909740ec9a34fd22: Deleting consensus metadata
I20260807 06:26:47.569218 11172 ts_tablet_manager.cc:1918] T 7ea880b202b8442fbaa96f9317818fdd P e07905770bda4a068f7ffdae24cbd65d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:26:47.569337 11172 log.cc:1196] T 7ea880b202b8442fbaa96f9317818fdd P e07905770bda4a068f7ffdae24cbd65d: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/7ea880b202b8442fbaa96f9317818fdd
I20260807 06:26:47.569624 11172 ts_tablet_manager.cc:1939] T 7ea880b202b8442fbaa96f9317818fdd P e07905770bda4a068f7ffdae24cbd65d: Deleting consensus metadata
I20260807 06:26:47.569880 10481 catalog_manager.cc:4841] TS 3b743124e345452c909740ec9a34fd22 (127.15.254.3:36049): tablet 7ea880b202b8442fbaa96f9317818fdd (table test-table [id=d24e426398824f9e990b5cee51552503]) successfully deleted
I20260807 06:26:47.570755 10481 catalog_manager.cc:4841] TS e07905770bda4a068f7ffdae24cbd65d (127.15.254.2:45383): tablet 7ea880b202b8442fbaa96f9317818fdd (table test-table [id=d24e426398824f9e990b5cee51552503]) successfully deleted
I20260807 06:26:47.575536 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 10759
W20260807 06:26:47.575582 10482 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.254.1:43821 (error 108)
I20260807 06:26:47.584286 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:47.583834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:43821 (local address 127.0.0.1:49018)
0807 06:26:47.584074 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:47.584164 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.1:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260807 06:26:47.584518 10482 catalog_manager.cc:4584] TS 8fd329ad1c78481395902376ffa76564 (127.15.254.1:43821): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7ea880b202b8442fbaa96f9317818fdd: Network error: Client connection negotiation failed: client connection to 127.15.254.1:43821: connect: Connection refused (error 111)
I20260807 06:26:47.593786 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 10939
I20260807 06:26:47.607069 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 10459
2026-08-07T06:26:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1333 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:26:47 dist-test-slave-44d2 krb5kdc[11176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:26:47 dist-test-slave-44d2 krb5kdc[11176](info): set up 2 sockets
Aug 07 06:26:47 dist-test-slave-44d2 krb5kdc[11176](info): commencing operation
krb5kdc: starting...
W20260807 06:26:48.969949 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.332s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:26:49 dist-test-slave-44d2 krb5kdc[11176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:26:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:26:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:26:49.111338 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:45277
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:45277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:26:49.227499 11201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:49.227761 11201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:49.227802 11201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:49.231668 11201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:26:49.231727 11201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:49.231746 11201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:49.231762 11201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:26:49.231777 11201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:26:49.236922 11201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:45277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:45277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:49.238001 11201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:49.239008 11201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:49.247638 11210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:49.247638 11214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:49.247810 11201 server_base.cc:1034] running on GCE node
W20260807 06:26:49.247730 11212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:49.248582 11201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:49.249724 11201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:49.250908 11201 hybrid_clock.cc:648] HybridClock initialized: now 1786084009250856 us; error 71 us; skew 500 ppm
Aug 07 06:26:49 dist-test-slave-44d2 krb5kdc[11176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084009, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:26:49.253942 11201 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:26:49.254908 11201 webserver.cc:460] Webserver started at http://127.15.254.62:34643/ using document root <none> and password file <none>
I20260807 06:26:49.255518 11201 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:49.255611 11201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:49.255852 11201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:49.258095 11201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "b1caf29e46c2426ebe4281c6d58ed68e"
format_stamp: "Formatted at 2026-08-07 06:26:49 on dist-test-slave-44d2"
server_key: "5311afd68367b59f4c7762b0cc67d89d"
server_key_iv: "66acbd4eb7d4bd2cf348a4177948be47"
server_key_version: "encryptionkey@0"
I20260807 06:26:49.258687 11201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "b1caf29e46c2426ebe4281c6d58ed68e"
format_stamp: "Formatted at 2026-08-07 06:26:49 on dist-test-slave-44d2"
server_key: "5311afd68367b59f4c7762b0cc67d89d"
server_key_iv: "66acbd4eb7d4bd2cf348a4177948be47"
server_key_version: "encryptionkey@0"
I20260807 06:26:49.262544 11201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:26:49.265169 11221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:49.266402 11201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 06:26:49.266564 11201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "b1caf29e46c2426ebe4281c6d58ed68e"
format_stamp: "Formatted at 2026-08-07 06:26:49 on dist-test-slave-44d2"
server_key: "5311afd68367b59f4c7762b0cc67d89d"
server_key_iv: "66acbd4eb7d4bd2cf348a4177948be47"
server_key_version: "encryptionkey@0"
I20260807 06:26:49.266681 11201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:49.281883 11201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:49.285338 11201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:49.285565 11201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:49.293144 11201 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:45277
I20260807 06:26:49.293167 11302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:45277 every 8 connection(s)
I20260807 06:26:49.294126 11201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:26:49.297075 11303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:49.298611 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 11201
I20260807 06:26:49.298769 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:26:49.299125 16376 external_mini_cluster.cc:1428] Setting key 793b85fca94d9fb5665d489ae64df2b7
I20260807 06:26:49.303534 11303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1caf29e46c2426ebe4281c6d58ed68e: Bootstrap starting.
I20260807 06:26:49.305895 11303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1caf29e46c2426ebe4281c6d58ed68e: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:49.306773 11303 log.cc:824] T 00000000000000000000000000000000 P b1caf29e46c2426ebe4281c6d58ed68e: Log is configured to *not* fsync() on all Append() calls
Aug 07 06:26:49 dist-test-slave-44d2 krb5kdc[11176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:26:49.309126 11303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1caf29e46c2426ebe4281c6d58ed68e: No bootstrap required, opened a new log
I20260807 06:26:49.313179 11303 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1caf29e46c2426ebe4281c6d58ed68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1caf29e46c2426ebe4281c6d58ed68e" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 45277 } }
I20260807 06:26:49.313407 11303 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1caf29e46c2426ebe4281c6d58ed68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:49.313467 11303 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1caf29e46c2426ebe4281c6d58ed68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1caf29e46c2426ebe4281c6d58ed68e, State: Initialized, Role: FOLLOWER
I20260807 06:26:49.313899 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:49.300672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55056 (local address 127.15.254.62:45277)
0807 06:26:49.301154 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:49.301169 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:49.301204 (+ 35us) server_negotiation.cc:407] Connection header received
0807 06:26:49.301838 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:49.301856 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:49.302186 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:49.302650 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:49.303461 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:49.304539 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:49.305403 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:49.305895 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:49.306100 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:49.308949 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:49.308955 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:49.308973 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:49.309007 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:49.310787 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:49.311327 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:49.311332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:49.311336 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:49.311443 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:49.311755 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:49.311757 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:49.311758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:49.312142 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:49.312295 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:49.312671 (+ 376us) server_negotiation.cc:299] Negotiation successful
0807 06:26:49.312995 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":149,"threads_started":1}
I20260807 06:26:49.313978 11303 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1caf29e46c2426ebe4281c6d58ed68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1caf29e46c2426ebe4281c6d58ed68e" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 45277 } }
I20260807 06:26:49.314154 11303 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1caf29e46c2426ebe4281c6d58ed68e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:26:49.314214 11303 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1caf29e46c2426ebe4281c6d58ed68e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:26:49.314281 11303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1caf29e46c2426ebe4281c6d58ed68e [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:49.315362 11303 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1caf29e46c2426ebe4281c6d58ed68e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1caf29e46c2426ebe4281c6d58ed68e" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 45277 } }
I20260807 06:26:49.315733 11303 leader_election.cc:302] T 00000000000000000000000000000000 P b1caf29e46c2426ebe4281c6d58ed68e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1caf29e46c2426ebe4281c6d58ed68e; no voters:
I20260807 06:26:49.316103 11303 leader_election.cc:288] T 00000000000000000000000000000000 P b1caf29e46c2426ebe4281c6d58ed68e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:26:49.316213 11309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1caf29e46c2426ebe4281c6d58ed68e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:26:49.316450 11309 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1caf29e46c2426ebe4281c6d58ed68e [term 1 LEADER]: Becoming Leader. State: Replica: b1caf29e46c2426ebe4281c6d58ed68e, State: Running, Role: LEADER
I20260807 06:26:49.316792 11309 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1caf29e46c2426ebe4281c6d58ed68e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1caf29e46c2426ebe4281c6d58ed68e" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 45277 } }
I20260807 06:26:49.317399 11303 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1caf29e46c2426ebe4281c6d58ed68e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:26:49.318514 11310 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1caf29e46c2426ebe4281c6d58ed68e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1caf29e46c2426ebe4281c6d58ed68e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1caf29e46c2426ebe4281c6d58ed68e" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 45277 } } }
I20260807 06:26:49.318652 11310 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1caf29e46c2426ebe4281c6d58ed68e [sys.catalog]: This master's current role is: LEADER
I20260807 06:26:49.318984 11311 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1caf29e46c2426ebe4281c6d58ed68e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1caf29e46c2426ebe4281c6d58ed68e. Latest consensus state: current_term: 1 leader_uuid: "b1caf29e46c2426ebe4281c6d58ed68e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1caf29e46c2426ebe4281c6d58ed68e" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 45277 } } }
I20260807 06:26:49.319077 11311 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1caf29e46c2426ebe4281c6d58ed68e [sys.catalog]: This master's current role is: LEADER
W20260807 06:26:49.322435 11330 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b1caf29e46c2426ebe4281c6d58ed68e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:26:49.322525 11330 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:26:49.322604 11321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:26:49.323274 11321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:26:49.329061 11321 catalog_manager.cc:1347] Generated new cluster ID: 42407a1d8f0645a5951ff151bc1295ca
I20260807 06:26:49.329137 11321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:26:49.352133 11321 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:26:49.353227 11321 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:26:49.365609 11321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1caf29e46c2426ebe4281c6d58ed68e: Generated new TSK 0
I20260807 06:26:49.366523 11321 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.254.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
Entry for principal kudu/127.15.254.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab.
I20260807 06:26:49.400956 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:0
--local_ip_for_outbound_sockets=127.15.254.1
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:45277
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:26:49.517439 11337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:49.517799 11337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:49.517910 11337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:49.522243 11337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:49.522311 11337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:49.522413 11337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:26:49.527791 11337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.1.keytab
--principal=kudu/127.15.254.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:45277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:49.528970 11337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:49.530030 11337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:49.540207 11343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:49.540212 11346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:49.540263 11337 server_base.cc:1034] running on GCE node
W20260807 06:26:49.540474 11344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:49.540896 11337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:49.541632 11337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:49.542819 11337 hybrid_clock.cc:648] HybridClock initialized: now 1786084009542801 us; error 29 us; skew 500 ppm
Aug 07 06:26:49 dist-test-slave-44d2 krb5kdc[11176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084009, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:26:49.545679 11337 init.cc:377] Logged in from keytab as kudu/127.15.254.1@KRBTEST.COM (short username kudu)
I20260807 06:26:49.546586 11337 webserver.cc:460] Webserver started at http://127.15.254.1:46721/ using document root <none> and password file <none>
I20260807 06:26:49.547173 11337 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:49.547266 11337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:49.547501 11337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:49.549671 11337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-0/data/instance:
uuid: "1df053d15c454f799d82a830ed2bdf1f"
format_stamp: "Formatted at 2026-08-07 06:26:49 on dist-test-slave-44d2"
server_key: "ab48e1d21923fa8830e89446eabcce25"
server_key_iv: "9ecbd7b8513aec5eeb298dcf5407f7c2"
server_key_version: "encryptionkey@0"
I20260807 06:26:49.550244 11337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance:
uuid: "1df053d15c454f799d82a830ed2bdf1f"
format_stamp: "Formatted at 2026-08-07 06:26:49 on dist-test-slave-44d2"
server_key: "ab48e1d21923fa8830e89446eabcce25"
server_key_iv: "9ecbd7b8513aec5eeb298dcf5407f7c2"
server_key_version: "encryptionkey@0"
I20260807 06:26:49.554113 11337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:26:49.556931 11359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:49.558183 11337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:26:49.558321 11337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "1df053d15c454f799d82a830ed2bdf1f"
format_stamp: "Formatted at 2026-08-07 06:26:49 on dist-test-slave-44d2"
server_key: "ab48e1d21923fa8830e89446eabcce25"
server_key_iv: "9ecbd7b8513aec5eeb298dcf5407f7c2"
server_key_version: "encryptionkey@0"
I20260807 06:26:49.558436 11337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:49.577514 11337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:49.580765 11337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:49.580984 11337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:49.581625 11337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:26:49.582523 11337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:26:49.582604 11337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:49.582693 11337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:26:49.582737 11337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:49.592017 11337 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:35729
I20260807 06:26:49.592048 11503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:35729 every 8 connection(s)
I20260807 06:26:49.592968 11337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:26:49.598902 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 11337
I20260807 06:26:49.599097 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance
I20260807 06:26:49.599459 16376 external_mini_cluster.cc:1428] Setting key 8162cbf83309d0a21ac2be6cc096e40f
Aug 07 06:26:49 dist-test-slave-44d2 krb5kdc[11176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084009, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.1@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:26:49.608314 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:49.594743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:36249 (local address 127.15.254.62:45277)
0807 06:26:49.594911 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:49.594916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:49.595759 (+ 843us) server_negotiation.cc:407] Connection header received
0807 06:26:49.597172 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:49.597175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:49.597225 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:49.597301 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:49.598523 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:49.599198 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:49.600702 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:49.601051 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:49.601167 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:49.604005 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:49.604011 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:49.604015 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:49.604050 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:49.605760 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:49.606469 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:49.606472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:49.606474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:49.606558 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:49.607025 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:49.607028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:49.607029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:49.607229 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:49.607318 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:49.608066 (+ 748us) server_negotiation.cc:299] Negotiation successful
0807 06:26:49.608183 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 06:26:49.609730 11507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:49.595150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:45277 (local address 127.15.254.1:36249)
0807 06:26:49.595609 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:49.595652 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:49.596933 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:49.597468 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:49.597475 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:49.597834 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:26:49.598362 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:49.598376 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:49.599349 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:49.599354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:49.600566 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:49.600577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:49.601212 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:49.601216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:49.601397 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:49.602025 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:26:49.602045 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:26:49.603741 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:26:49.605890 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:49.605900 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:49.605921 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:49.606341 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:49.606658 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:49.606664 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:49.606669 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:49.606880 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:49.607350 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:26:49.607357 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:26:49.607737 (+ 380us) client_negotiation.cc:769] Sending connection context
0807 06:26:49.608003 (+ 266us) client_negotiation.cc:240] Negotiation successful
0807 06:26:49.608436 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":123,"threads_started":1}
I20260807 06:26:49.611217 11505 heartbeater.cc:347] Connected to a master server at 127.15.254.62:45277
I20260807 06:26:49.611496 11505 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.15.254.2@KRBTEST.COM; defaulting to no policy
I20260807 06:26:49.612119 11505 heartbeater.cc:511] Master 127.15.254.62:45277 requested a full tablet report, sending...
I20260807 06:26:49.613797 11255 ts_manager.cc:194] Registered new tserver with Master: 1df053d15c454f799d82a830ed2bdf1f (127.15.254.1:35729)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.2@KRBTEST.COM" created.
I20260807 06:26:49.615111 11255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.1@KRBTEST.COM'} at 127.15.254.1:36249
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
Entry for principal kudu/127.15.254.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab.
I20260807 06:26:49.632248 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:0
--local_ip_for_outbound_sockets=127.15.254.2
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:45277
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:26:49.748265 11510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:49.748623 11510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:49.748674 11510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:49.752779 11510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:49.752846 11510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:49.752907 11510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:26:49.758279 11510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.2.keytab
--principal=kudu/127.15.254.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:45277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:49.759371 11510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:49.760370 11510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:49.770678 11522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:49.770709 11520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:49.770714 11519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:49.771158 11510 server_base.cc:1034] running on GCE node
I20260807 06:26:49.771646 11510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:49.772346 11510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:49.773564 11510 hybrid_clock.cc:648] HybridClock initialized: now 1786084009773535 us; error 40 us; skew 500 ppm
Aug 07 06:26:49 dist-test-slave-44d2 krb5kdc[11176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084009, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:26:49.776659 11510 init.cc:377] Logged in from keytab as kudu/127.15.254.2@KRBTEST.COM (short username kudu)
I20260807 06:26:49.777623 11510 webserver.cc:460] Webserver started at http://127.15.254.2:37411/ using document root <none> and password file <none>
I20260807 06:26:49.778231 11510 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:49.778329 11510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:49.778581 11510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:49.780853 11510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-1/data/instance:
uuid: "72af02523ac64ab88cf7ee88678ddff9"
format_stamp: "Formatted at 2026-08-07 06:26:49 on dist-test-slave-44d2"
server_key: "af7dc71e4a69b4fb29a4216af327c9f5"
server_key_iv: "76fd657a1de8428e6daa3af084663450"
server_key_version: "encryptionkey@0"
I20260807 06:26:49.781476 11510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance:
uuid: "72af02523ac64ab88cf7ee88678ddff9"
format_stamp: "Formatted at 2026-08-07 06:26:49 on dist-test-slave-44d2"
server_key: "af7dc71e4a69b4fb29a4216af327c9f5"
server_key_iv: "76fd657a1de8428e6daa3af084663450"
server_key_version: "encryptionkey@0"
I20260807 06:26:49.785745 11510 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260807 06:26:49.788434 11531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:49.789889 11510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 06:26:49.790051 11510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "72af02523ac64ab88cf7ee88678ddff9"
format_stamp: "Formatted at 2026-08-07 06:26:49 on dist-test-slave-44d2"
server_key: "af7dc71e4a69b4fb29a4216af327c9f5"
server_key_iv: "76fd657a1de8428e6daa3af084663450"
server_key_version: "encryptionkey@0"
I20260807 06:26:49.790172 11510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:49.804358 11510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:49.808059 11510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:49.808288 11510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:49.809008 11510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:26:49.809949 11510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:26:49.810038 11510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:49.810125 11510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:26:49.810170 11510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:49.819365 11510 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:39053
I20260807 06:26:49.819384 11685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:39053 every 8 connection(s)
I20260807 06:26:49.820408 11510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:26:49.829815 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 11510
I20260807 06:26:49.829982 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance
I20260807 06:26:49.830336 16376 external_mini_cluster.cc:1428] Setting key 8557ed3460439ed1038e0b40d90de3df
Aug 07 06:26:49 dist-test-slave-44d2 krb5kdc[11176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084009, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.2@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:26:49.835644 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:49.822296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:41249 (local address 127.15.254.62:45277)
0807 06:26:49.822466 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:49.822470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:49.823316 (+ 846us) server_negotiation.cc:407] Connection header received
0807 06:26:49.824704 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:49.824708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:49.824764 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:49.824864 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:49.826400 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:49.826936 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:49.828090 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:49.828316 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:49.828426 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:49.831208 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:49.831212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:49.831214 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:49.831239 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:49.833059 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:49.833573 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:49.833580 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:49.833584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:49.833685 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:49.834029 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:49.834036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:49.834040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:49.834309 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:49.834422 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:49.835189 (+ 767us) server_negotiation.cc:299] Negotiation successful
0807 06:26:49.835384 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 06:26:49.836632 11689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:49.822669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:45277 (local address 127.15.254.2:41249)
0807 06:26:49.823164 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:49.823210 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:49.824442 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:49.825097 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:49.825104 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:49.825631 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:26:49.826251 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:49.826266 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:49.827069 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:49.827073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:49.827951 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:49.827968 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:49.828469 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:49.828473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:49.828673 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:49.829283 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:26:49.829299 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:26:49.831017 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:26:49.833180 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:49.833185 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:49.833198 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:49.833471 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:49.833779 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:49.833782 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:49.833783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:49.833933 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:49.834434 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:26:49.834443 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:26:49.834835 (+ 392us) client_negotiation.cc:769] Sending connection context
0807 06:26:49.835066 (+ 231us) client_negotiation.cc:240] Negotiation successful
0807 06:26:49.835452 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":126,"threads_started":1}
I20260807 06:26:49.838130 11687 heartbeater.cc:347] Connected to a master server at 127.15.254.62:45277
I20260807 06:26:49.838426 11687 heartbeater.cc:464] Registering TS with master...
I20260807 06:26:49.839020 11687 heartbeater.cc:511] Master 127.15.254.62:45277 requested a full tablet report, sending...
I20260807 06:26:49.840153 11255 ts_manager.cc:194] Registered new tserver with Master: 72af02523ac64ab88cf7ee88678ddff9 (127.15.254.2:39053)
I20260807 06:26:49.840788 11255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.2@KRBTEST.COM'} at 127.15.254.2:41249
WARNING: no policy specified for kudu/127.15.254.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
Entry for principal kudu/127.15.254.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab.
I20260807 06:26:49.863741 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:0
--local_ip_for_outbound_sockets=127.15.254.3
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:45277
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:26:49.981065 11697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:49.981335 11697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:49.981401 11697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:49.985476 11697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:49.985541 11697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:49.985626 11697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:26:49.990994 11697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.3.keytab
--principal=kudu/127.15.254.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:45277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:49.992063 11697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:49.993165 11697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:50.003600 11707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:50.003592 11704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:50.003592 11703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:50.003738 11697 server_base.cc:1034] running on GCE node
I20260807 06:26:50.004855 11697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:50.005673 11697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:50.006947 11697 hybrid_clock.cc:648] HybridClock initialized: now 1786084010006931 us; error 38 us; skew 500 ppm
Aug 07 06:26:50 dist-test-slave-44d2 krb5kdc[11176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084010, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:26:50.010635 11697 init.cc:377] Logged in from keytab as kudu/127.15.254.3@KRBTEST.COM (short username kudu)
I20260807 06:26:50.011720 11697 webserver.cc:460] Webserver started at http://127.15.254.3:32845/ using document root <none> and password file <none>
I20260807 06:26:50.012404 11697 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:50.012555 11697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:50.012838 11697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:50.015223 11697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-2/data/instance:
uuid: "169c3540e3e04f7c84736ae7475014ab"
format_stamp: "Formatted at 2026-08-07 06:26:50 on dist-test-slave-44d2"
server_key: "24b62dfb805fee3362231de49f4215d3"
server_key_iv: "e02d8ca06568ab1f2a740ffea3c9b4ed"
server_key_version: "encryptionkey@0"
I20260807 06:26:50.015863 11697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance:
uuid: "169c3540e3e04f7c84736ae7475014ab"
format_stamp: "Formatted at 2026-08-07 06:26:50 on dist-test-slave-44d2"
server_key: "24b62dfb805fee3362231de49f4215d3"
server_key_iv: "e02d8ca06568ab1f2a740ffea3c9b4ed"
server_key_version: "encryptionkey@0"
I20260807 06:26:50.020380 11697 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260807 06:26:50.023146 11716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:50.024587 11697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:26:50.024721 11697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "169c3540e3e04f7c84736ae7475014ab"
format_stamp: "Formatted at 2026-08-07 06:26:50 on dist-test-slave-44d2"
server_key: "24b62dfb805fee3362231de49f4215d3"
server_key_iv: "e02d8ca06568ab1f2a740ffea3c9b4ed"
server_key_version: "encryptionkey@0"
I20260807 06:26:50.024833 11697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:50.058859 11697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:50.062320 11697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:50.062542 11697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:50.063211 11697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:26:50.064127 11697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:26:50.064198 11697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:50.064272 11697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:26:50.064313 11697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:50.074232 11697 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:42665
I20260807 06:26:50.074255 11866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:42665 every 8 connection(s)
I20260807 06:26:50.075184 11697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:26:50.081683 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 11697
I20260807 06:26:50.081872 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance
I20260807 06:26:50.082270 16376 external_mini_cluster.cc:1428] Setting key 0e9c07d1aa75c419480937ceb5683ff9
Aug 07 06:26:50 dist-test-slave-44d2 krb5kdc[11176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084010, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.3@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:26:50.091518 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:50.077216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:42063 (local address 127.15.254.62:45277)
0807 06:26:50.077376 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:50.077381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:50.078358 (+ 977us) server_negotiation.cc:407] Connection header received
0807 06:26:50.079646 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:50.079649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:50.079695 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:50.079772 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:50.081124 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.081667 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.083143 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.083459 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.083573 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:50.086899 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:50.086960 (+ 61us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:50.086963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:50.086994 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:50.088879 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:50.089563 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:50.089567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:50.089570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:50.089673 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:50.090125 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:50.090130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:50.090132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:50.090372 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:50.090497 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:50.091148 (+ 651us) server_negotiation.cc:299] Negotiation successful
0807 06:26:50.091319 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 06:26:50.092808 11871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:50.077664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:45277 (local address 127.15.254.3:42063)
0807 06:26:50.078194 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:50.078240 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:50.079433 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:50.079943 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:50.079950 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:50.080365 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:26:50.080972 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:50.080994 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.081809 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.081814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:50.082988 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:50.082999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.083621 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.083626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:50.083854 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:50.084852 (+ 998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:26:50.084870 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:26:50.086672 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:26:50.089038 (+ 2366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:50.089047 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:50.089071 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:50.089437 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:50.089785 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:26:50.089789 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:26:50.089791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:26:50.090014 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:26:50.090500 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:26:50.090511 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:26:50.090857 (+ 346us) client_negotiation.cc:769] Sending connection context
0807 06:26:50.091099 (+ 242us) client_negotiation.cc:240] Negotiation successful
0807 06:26:50.091572 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":131,"threads_started":1}
I20260807 06:26:50.094465 11868 heartbeater.cc:347] Connected to a master server at 127.15.254.62:45277
I20260807 06:26:50.094925 11868 heartbeater.cc:464] Registering TS with master...
I20260807 06:26:50.095779 11868 heartbeater.cc:511] Master 127.15.254.62:45277 requested a full tablet report, sending...
I20260807 06:26:50.097078 11255 ts_manager.cc:194] Registered new tserver with Master: 169c3540e3e04f7c84736ae7475014ab (127.15.254.3:42665)
I20260807 06:26:50.097718 11255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.254.3@KRBTEST.COM'} at 127.15.254.3:42063
I20260807 06:26:50.108947 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:26:50.118741 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:50.111072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55060 (local address 127.15.254.62:45277)
0807 06:26:50.111234 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:50.111238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:50.111427 (+ 189us) server_negotiation.cc:407] Connection header received
0807 06:26:50.111472 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:50.111475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:50.111518 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:50.111591 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:50.112285 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.112932 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.113727 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.113971 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.114054 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:50.115438 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:50.115442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:50.115443 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:50.115468 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:50.116955 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:50.117382 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:50.117385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:50.117386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:50.117475 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:50.117701 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:50.117703 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:50.117704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:50.118028 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:50.118180 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:50.118452 (+ 272us) server_negotiation.cc:299] Negotiation successful
0807 06:26:50.118573 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260807 06:26:50.121877 11255 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:26:50.125015 11255 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:26:50.139557 11889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:50.134114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:35729 (local address 127.0.0.1:56902)
0807 06:26:50.134772 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:50.134785 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:50.134884 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:50.135390 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:50.135393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:50.135413 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:50.135550 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:50.135559 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.137076 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.137079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:50.138151 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:50.138161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.138996 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.139000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:50.139110 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:50.139153 (+ 43us) client_negotiation.cc:769] Sending connection context
0807 06:26:50.139271 (+ 118us) client_negotiation.cc:240] Negotiation successful
0807 06:26:50.139394 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":68,"thread_start_us":101,"threads_started":1}
I20260807 06:26:50.139580 11890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:50.134298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56902 (local address 127.15.254.1:35729)
0807 06:26:50.134657 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:50.134663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:50.134990 (+ 327us) server_negotiation.cc:407] Connection header received
0807 06:26:50.135076 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:50.135081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:50.135251 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:50.135395 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:50.135687 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.136914 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.138269 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.138857 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.138973 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:50.139124 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:50.139329 (+ 205us) server_negotiation.cc:299] Negotiation successful
0807 06:26:50.139429 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":132,"threads_started":1}
I20260807 06:26:50.140161 11886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:50.133333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:42665 (local address 127.0.0.1:34042)
0807 06:26:50.134430 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:50.134442 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:50.134569 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:50.135318 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:50.135326 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:50.135353 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:50.135499 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:50.135508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.136614 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.136618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:50.138206 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:50.138217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.139529 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.139533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:50.139632 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:50.139879 (+ 247us) client_negotiation.cc:769] Sending connection context
0807 06:26:50.139952 (+ 73us) client_negotiation.cc:240] Negotiation successful
0807 06:26:50.140020 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1047,"mutex_wait_us":170,"thread_start_us":98,"threads_started":1}
I20260807 06:26:50.140340 11884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:50.133333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:39053 (local address 127.0.0.1:55262)
0807 06:26:50.133700 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:50.133750 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:50.133873 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:50.135596 (+ 1723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:50.135599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:50.135618 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:50.135707 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:50.135712 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.137096 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.137099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:50.138731 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:50.138741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.139961 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.139965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:50.140058 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:50.140080 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 06:26:50.140145 (+ 65us) client_negotiation.cc:240] Negotiation successful
0807 06:26:50.140220 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":457,"thread_start_us":90,"threads_started":1}
I20260807 06:26:50.140722 11887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:50.133462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34042 (local address 127.15.254.3:42665)
0807 06:26:50.134306 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:50.134319 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:50.134666 (+ 347us) server_negotiation.cc:407] Connection header received
0807 06:26:50.134753 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:50.134758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:50.134964 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:50.135122 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:50.135630 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.136453 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.138349 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.139226 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.139322 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:50.139454 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:50.140328 (+ 874us) server_negotiation.cc:299] Negotiation successful
0807 06:26:50.140537 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":154,"threads_started":1}
I20260807 06:26:50.142004 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:50.134038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.15.254.2:39053)
0807 06:26:50.135143 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:50.135153 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:50.135171 (+ 18us) server_negotiation.cc:407] Connection header received
0807 06:26:50.135233 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:50.135237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:50.135401 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:50.135605 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:50.135813 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.136925 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.138861 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.139856 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.139962 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:50.140093 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:50.140183 (+ 90us) server_negotiation.cc:299] Negotiation successful
0807 06:26:50.141858 (+ 1675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":984,"thread_start_us":145,"threads_started":1}
I20260807 06:26:50.142632 11420 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb450e8229d46bcb8e1d6867d52e931 (DEFAULT_TABLE table=test-table [id=9dd12112d46742ef9c9f7d7b2a3cefb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:26:50.143437 11781 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb450e8229d46bcb8e1d6867d52e931 (DEFAULT_TABLE table=test-table [id=9dd12112d46742ef9c9f7d7b2a3cefb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:26:50.143851 11420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb450e8229d46bcb8e1d6867d52e931. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:50.144340 11781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb450e8229d46bcb8e1d6867d52e931. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:50.144635 11600 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb450e8229d46bcb8e1d6867d52e931 (DEFAULT_TABLE table=test-table [id=9dd12112d46742ef9c9f7d7b2a3cefb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:26:50.145633 11600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb450e8229d46bcb8e1d6867d52e931. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:50.150473 11891 tablet_bootstrap.cc:492] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab: Bootstrap starting.
I20260807 06:26:50.152848 11893 tablet_bootstrap.cc:492] T 9cb450e8229d46bcb8e1d6867d52e931 P 1df053d15c454f799d82a830ed2bdf1f: Bootstrap starting.
I20260807 06:26:50.153010 11891 tablet_bootstrap.cc:654] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:50.153453 11895 tablet_bootstrap.cc:492] T 9cb450e8229d46bcb8e1d6867d52e931 P 72af02523ac64ab88cf7ee88678ddff9: Bootstrap starting.
I20260807 06:26:50.154583 11891 log.cc:824] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:50.154866 11893 tablet_bootstrap.cc:654] T 9cb450e8229d46bcb8e1d6867d52e931 P 1df053d15c454f799d82a830ed2bdf1f: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:50.155642 11893 log.cc:824] T 9cb450e8229d46bcb8e1d6867d52e931 P 1df053d15c454f799d82a830ed2bdf1f: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:50.155946 11895 tablet_bootstrap.cc:654] T 9cb450e8229d46bcb8e1d6867d52e931 P 72af02523ac64ab88cf7ee88678ddff9: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:50.156761 11891 tablet_bootstrap.cc:492] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab: No bootstrap required, opened a new log
I20260807 06:26:50.156769 11895 log.cc:824] T 9cb450e8229d46bcb8e1d6867d52e931 P 72af02523ac64ab88cf7ee88678ddff9: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:50.156980 11891 ts_tablet_manager.cc:1397] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260807 06:26:50.157701 11893 tablet_bootstrap.cc:492] T 9cb450e8229d46bcb8e1d6867d52e931 P 1df053d15c454f799d82a830ed2bdf1f: No bootstrap required, opened a new log
I20260807 06:26:50.157886 11893 ts_tablet_manager.cc:1397] T 9cb450e8229d46bcb8e1d6867d52e931 P 1df053d15c454f799d82a830ed2bdf1f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260807 06:26:50.158800 11895 tablet_bootstrap.cc:492] T 9cb450e8229d46bcb8e1d6867d52e931 P 72af02523ac64ab88cf7ee88678ddff9: No bootstrap required, opened a new log
I20260807 06:26:50.158998 11895 ts_tablet_manager.cc:1397] T 9cb450e8229d46bcb8e1d6867d52e931 P 72af02523ac64ab88cf7ee88678ddff9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 06:26:50.161114 11891 raft_consensus.cc:348] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c3540e3e04f7c84736ae7475014ab" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 42665 } } peers { permanent_uuid: "72af02523ac64ab88cf7ee88678ddff9" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39053 } } peers { permanent_uuid: "1df053d15c454f799d82a830ed2bdf1f" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35729 } }
I20260807 06:26:50.161412 11891 raft_consensus.cc:374] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:50.161470 11891 raft_consensus.cc:729] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 169c3540e3e04f7c84736ae7475014ab, State: Initialized, Role: FOLLOWER
I20260807 06:26:50.161898 11895 raft_consensus.cc:348] T 9cb450e8229d46bcb8e1d6867d52e931 P 72af02523ac64ab88cf7ee88678ddff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c3540e3e04f7c84736ae7475014ab" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 42665 } } peers { permanent_uuid: "72af02523ac64ab88cf7ee88678ddff9" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39053 } } peers { permanent_uuid: "1df053d15c454f799d82a830ed2bdf1f" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35729 } }
I20260807 06:26:50.162148 11895 raft_consensus.cc:374] T 9cb450e8229d46bcb8e1d6867d52e931 P 72af02523ac64ab88cf7ee88678ddff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:50.162204 11895 raft_consensus.cc:729] T 9cb450e8229d46bcb8e1d6867d52e931 P 72af02523ac64ab88cf7ee88678ddff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72af02523ac64ab88cf7ee88678ddff9, State: Initialized, Role: FOLLOWER
I20260807 06:26:50.162101 11891 consensus_queue.cc:260] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c3540e3e04f7c84736ae7475014ab" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 42665 } } peers { permanent_uuid: "72af02523ac64ab88cf7ee88678ddff9" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39053 } } peers { permanent_uuid: "1df053d15c454f799d82a830ed2bdf1f" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35729 } }
I20260807 06:26:50.162340 11893 raft_consensus.cc:348] T 9cb450e8229d46bcb8e1d6867d52e931 P 1df053d15c454f799d82a830ed2bdf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c3540e3e04f7c84736ae7475014ab" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 42665 } } peers { permanent_uuid: "72af02523ac64ab88cf7ee88678ddff9" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39053 } } peers { permanent_uuid: "1df053d15c454f799d82a830ed2bdf1f" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35729 } }
I20260807 06:26:50.162576 11893 raft_consensus.cc:374] T 9cb450e8229d46bcb8e1d6867d52e931 P 1df053d15c454f799d82a830ed2bdf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:50.162671 11893 raft_consensus.cc:729] T 9cb450e8229d46bcb8e1d6867d52e931 P 1df053d15c454f799d82a830ed2bdf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df053d15c454f799d82a830ed2bdf1f, State: Initialized, Role: FOLLOWER
I20260807 06:26:50.162703 11895 consensus_queue.cc:260] T 9cb450e8229d46bcb8e1d6867d52e931 P 72af02523ac64ab88cf7ee88678ddff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c3540e3e04f7c84736ae7475014ab" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 42665 } } peers { permanent_uuid: "72af02523ac64ab88cf7ee88678ddff9" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39053 } } peers { permanent_uuid: "1df053d15c454f799d82a830ed2bdf1f" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35729 } }
I20260807 06:26:50.163137 11893 consensus_queue.cc:260] T 9cb450e8229d46bcb8e1d6867d52e931 P 1df053d15c454f799d82a830ed2bdf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c3540e3e04f7c84736ae7475014ab" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 42665 } } peers { permanent_uuid: "72af02523ac64ab88cf7ee88678ddff9" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39053 } } peers { permanent_uuid: "1df053d15c454f799d82a830ed2bdf1f" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35729 } }
I20260807 06:26:50.163786 11895 ts_tablet_manager.cc:1428] T 9cb450e8229d46bcb8e1d6867d52e931 P 72af02523ac64ab88cf7ee88678ddff9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260807 06:26:50.163817 11891 ts_tablet_manager.cc:1428] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260807 06:26:50.163983 11893 ts_tablet_manager.cc:1428] T 9cb450e8229d46bcb8e1d6867d52e931 P 1df053d15c454f799d82a830ed2bdf1f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260807 06:26:50.164166 11687 heartbeater.cc:503] Master 127.15.254.62:45277 was elected leader, sending a full tablet report...
I20260807 06:26:50.164170 11868 heartbeater.cc:503] Master 127.15.254.62:45277 was elected leader, sending a full tablet report...
I20260807 06:26:50.164254 11505 heartbeater.cc:503] Master 127.15.254.62:45277 was elected leader, sending a full tablet report...
I20260807 06:26:50.265727 11900 raft_consensus.cc:482] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:26:50.265983 11900 raft_consensus.cc:504] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c3540e3e04f7c84736ae7475014ab" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 42665 } } peers { permanent_uuid: "72af02523ac64ab88cf7ee88678ddff9" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39053 } } peers { permanent_uuid: "1df053d15c454f799d82a830ed2bdf1f" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35729 } }
I20260807 06:26:50.267299 11900 leader_election.cc:288] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72af02523ac64ab88cf7ee88678ddff9 (127.15.254.2:39053), 1df053d15c454f799d82a830ed2bdf1f (127.15.254.1:35729)
I20260807 06:26:50.271205 11871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:50.267604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:35729 (local address 127.15.254.3:41803)
0807 06:26:50.267774 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:50.267812 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:50.267927 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:50.268182 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:50.268185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:50.268202 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:50.268330 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:50.268336 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.269127 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.269131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:50.270014 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:50.270025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.270764 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.270768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:50.270874 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:50.270900 (+ 26us) client_negotiation.cc:769] Sending connection context
0807 06:26:50.270986 (+ 86us) client_negotiation.cc:240] Negotiation successful
0807 06:26:50.271062 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260807 06:26:50.271205 11890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:50.267761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:41803 (local address 127.15.254.1:35729)
0807 06:26:50.267909 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:50.267914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:50.267932 (+ 18us) server_negotiation.cc:407] Connection header received
0807 06:26:50.268032 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:50.268035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:50.268084 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:50.268168 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:50.268444 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.269011 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.270135 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.270642 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.270769 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:50.270825 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:50.271005 (+ 180us) server_negotiation.cc:299] Negotiation successful
0807 06:26:50.271099 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 06:26:50.271996 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:50.267620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:39053 (local address 127.15.254.3:33299)
0807 06:26:50.268294 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:50.268306 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:50.268454 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:50.268770 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:50.268773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:50.268788 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:50.268926 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:50.268932 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.269649 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.269655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:50.270588 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:50.270598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.271550 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.271554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:50.271680 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:50.271706 (+ 26us) client_negotiation.cc:769] Sending connection context
0807 06:26:50.271784 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 06:26:50.271861 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":199,"thread_start_us":153,"threads_started":1}
I20260807 06:26:50.272024 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:50.267713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:33299 (local address 127.15.254.2:39053)
0807 06:26:50.267856 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:50.267862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:50.268422 (+ 560us) server_negotiation.cc:407] Connection header received
0807 06:26:50.268629 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:50.268631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:50.268678 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:50.268763 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:50.269042 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.269536 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.270720 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.271290 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.271429 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:50.271474 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:50.271826 (+ 352us) server_negotiation.cc:299] Negotiation successful
0807 06:26:50.271894 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 06:26:50.272027 11443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb450e8229d46bcb8e1d6867d52e931" candidate_uuid: "169c3540e3e04f7c84736ae7475014ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df053d15c454f799d82a830ed2bdf1f" is_pre_election: true
I20260807 06:26:50.272411 11443 raft_consensus.cc:2454] T 9cb450e8229d46bcb8e1d6867d52e931 P 1df053d15c454f799d82a830ed2bdf1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 169c3540e3e04f7c84736ae7475014ab in term 0.
I20260807 06:26:50.272725 11625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb450e8229d46bcb8e1d6867d52e931" candidate_uuid: "169c3540e3e04f7c84736ae7475014ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72af02523ac64ab88cf7ee88678ddff9" is_pre_election: true
I20260807 06:26:50.273000 11721 leader_election.cc:302] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 169c3540e3e04f7c84736ae7475014ab, 1df053d15c454f799d82a830ed2bdf1f; no voters:
I20260807 06:26:50.273149 11625 raft_consensus.cc:2454] T 9cb450e8229d46bcb8e1d6867d52e931 P 72af02523ac64ab88cf7ee88678ddff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 169c3540e3e04f7c84736ae7475014ab in term 0.
I20260807 06:26:50.273274 11900 raft_consensus.cc:2781] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:26:50.273367 11900 raft_consensus.cc:482] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:26:50.273569 11900 raft_consensus.cc:3037] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:50.274693 11900 raft_consensus.cc:504] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c3540e3e04f7c84736ae7475014ab" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 42665 } } peers { permanent_uuid: "72af02523ac64ab88cf7ee88678ddff9" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39053 } } peers { permanent_uuid: "1df053d15c454f799d82a830ed2bdf1f" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35729 } }
I20260807 06:26:50.275084 11900 leader_election.cc:288] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab [CANDIDATE]: Term 1 election: Requested vote from peers 72af02523ac64ab88cf7ee88678ddff9 (127.15.254.2:39053), 1df053d15c454f799d82a830ed2bdf1f (127.15.254.1:35729)
I20260807 06:26:50.275463 11443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb450e8229d46bcb8e1d6867d52e931" candidate_uuid: "169c3540e3e04f7c84736ae7475014ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df053d15c454f799d82a830ed2bdf1f"
I20260807 06:26:50.275508 11625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb450e8229d46bcb8e1d6867d52e931" candidate_uuid: "169c3540e3e04f7c84736ae7475014ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72af02523ac64ab88cf7ee88678ddff9"
I20260807 06:26:50.275594 11443 raft_consensus.cc:3037] T 9cb450e8229d46bcb8e1d6867d52e931 P 1df053d15c454f799d82a830ed2bdf1f [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:50.275640 11625 raft_consensus.cc:3037] T 9cb450e8229d46bcb8e1d6867d52e931 P 72af02523ac64ab88cf7ee88678ddff9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:50.276854 11625 raft_consensus.cc:2454] T 9cb450e8229d46bcb8e1d6867d52e931 P 72af02523ac64ab88cf7ee88678ddff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 169c3540e3e04f7c84736ae7475014ab in term 1.
I20260807 06:26:50.276873 11443 raft_consensus.cc:2454] T 9cb450e8229d46bcb8e1d6867d52e931 P 1df053d15c454f799d82a830ed2bdf1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 169c3540e3e04f7c84736ae7475014ab in term 1.
I20260807 06:26:50.277213 11717 leader_election.cc:302] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 169c3540e3e04f7c84736ae7475014ab, 72af02523ac64ab88cf7ee88678ddff9; no voters:
I20260807 06:26:50.277472 11900 raft_consensus.cc:2781] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:26:50.277717 11900 raft_consensus.cc:686] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab [term 1 LEADER]: Becoming Leader. State: Replica: 169c3540e3e04f7c84736ae7475014ab, State: Running, Role: LEADER
I20260807 06:26:50.278087 11900 consensus_queue.cc:237] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c3540e3e04f7c84736ae7475014ab" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 42665 } } peers { permanent_uuid: "72af02523ac64ab88cf7ee88678ddff9" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39053 } } peers { permanent_uuid: "1df053d15c454f799d82a830ed2bdf1f" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35729 } }
I20260807 06:26:50.281392 11254 catalog_manager.cc:5495] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 169c3540e3e04f7c84736ae7475014ab (127.15.254.3). New cstate: current_term: 1 leader_uuid: "169c3540e3e04f7c84736ae7475014ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c3540e3e04f7c84736ae7475014ab" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 42665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72af02523ac64ab88cf7ee88678ddff9" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1df053d15c454f799d82a830ed2bdf1f" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35729 } health_report { overall_health: UNKNOWN } } }
I20260807 06:26:50.296622 11887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:50.292455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44512 (local address 127.15.254.3:42665)
0807 06:26:50.292650 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:50.292653 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:50.292669 (+ 16us) server_negotiation.cc:407] Connection header received
0807 06:26:50.292793 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:50.292796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:50.292858 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:50.293002 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:26:50.293298 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.293798 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.295075 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.295373 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.295455 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:50.295783 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:26:50.296179 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:26:50.296272 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:50.296372 (+ 100us) server_negotiation.cc:299] Negotiation successful
0807 06:26:50.296425 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260807 06:26:50.302610 11625 raft_consensus.cc:1261] T 9cb450e8229d46bcb8e1d6867d52e931 P 72af02523ac64ab88cf7ee88678ddff9 [term 1 FOLLOWER]: Refusing update from remote peer 169c3540e3e04f7c84736ae7475014ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:26:50.302668 11443 raft_consensus.cc:1261] T 9cb450e8229d46bcb8e1d6867d52e931 P 1df053d15c454f799d82a830ed2bdf1f [term 1 FOLLOWER]: Refusing update from remote peer 169c3540e3e04f7c84736ae7475014ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:26:50.303383 11908 consensus_queue.cc:1035] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "72af02523ac64ab88cf7ee88678ddff9" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 39053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:26:50.303588 11900 consensus_queue.cc:1035] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df053d15c454f799d82a830ed2bdf1f" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:26:50.320739 11254 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:26:50.320899 11254 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:26:50.323534 11254 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1caf29e46c2426ebe4281c6d58ed68e: Sending DeleteTablet for 3 replicas of tablet 9cb450e8229d46bcb8e1d6867d52e931
I20260807 06:26:50.324383 11781 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cb450e8229d46bcb8e1d6867d52e931 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:26:50 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:34042
I20260807 06:26:50.324383 11600 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cb450e8229d46bcb8e1d6867d52e931 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:26:50 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:55262
I20260807 06:26:50.324441 11420 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cb450e8229d46bcb8e1d6867d52e931 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:26:50 UTC) from {username='kudu', principal='kudu/127.15.254.62@KRBTEST.COM'} at 127.0.0.1:56902
I20260807 06:26:50.324931 11928 tablet_replica.cc:330] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab: stopping tablet replica
I20260807 06:26:50.325255 11928 raft_consensus.cc:2229] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:26:50.325661 11928 raft_consensus.cc:2258] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:26:50.325907 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 11337
I20260807 06:26:50.334046 11928 ts_tablet_manager.cc:1905] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:26:50.334586 11929 tablet_replica.cc:330] T 9cb450e8229d46bcb8e1d6867d52e931 P 72af02523ac64ab88cf7ee88678ddff9: stopping tablet replica
I20260807 06:26:50.334969 11929 raft_consensus.cc:2229] T 9cb450e8229d46bcb8e1d6867d52e931 P 72af02523ac64ab88cf7ee88678ddff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:26:50.335412 11929 raft_consensus.cc:2258] T 9cb450e8229d46bcb8e1d6867d52e931 P 72af02523ac64ab88cf7ee88678ddff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:26:50.336714 11928 ts_tablet_manager.cc:1918] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:26:50.336833 11928 log.cc:1196] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/9cb450e8229d46bcb8e1d6867d52e931
I20260807 06:26:50.337081 11929 ts_tablet_manager.cc:1905] T 9cb450e8229d46bcb8e1d6867d52e931 P 72af02523ac64ab88cf7ee88678ddff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:26:50.337182 11928 ts_tablet_manager.cc:1939] T 9cb450e8229d46bcb8e1d6867d52e931 P 169c3540e3e04f7c84736ae7475014ab: Deleting consensus metadata
I20260807 06:26:50.338397 11230 catalog_manager.cc:4841] TS 169c3540e3e04f7c84736ae7475014ab (127.15.254.3:42665): tablet 9cb450e8229d46bcb8e1d6867d52e931 (table test-table [id=9dd12112d46742ef9c9f7d7b2a3cefb4]) successfully deleted
I20260807 06:26:50.339385 11929 ts_tablet_manager.cc:1918] T 9cb450e8229d46bcb8e1d6867d52e931 P 72af02523ac64ab88cf7ee88678ddff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:26:50.339476 11929 log.cc:1196] T 9cb450e8229d46bcb8e1d6867d52e931 P 72af02523ac64ab88cf7ee88678ddff9: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/9cb450e8229d46bcb8e1d6867d52e931
I20260807 06:26:50.339753 11929 ts_tablet_manager.cc:1939] T 9cb450e8229d46bcb8e1d6867d52e931 P 72af02523ac64ab88cf7ee88678ddff9: Deleting consensus metadata
I20260807 06:26:50.340870 11224 catalog_manager.cc:4841] TS 72af02523ac64ab88cf7ee88678ddff9 (127.15.254.2:39053): tablet 9cb450e8229d46bcb8e1d6867d52e931 (table test-table [id=9dd12112d46742ef9c9f7d7b2a3cefb4]) successfully deleted
W20260807 06:26:50.347090 11230 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.254.1:35729 (error 108)
I20260807 06:26:50.347236 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 11510
I20260807 06:26:50.356254 11884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:50.355928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:35729 (local address 127.0.0.1:54668)
0807 06:26:50.356075 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:50.356163 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.1:35729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 06:26:50.356720 11230 catalog_manager.cc:4584] TS 1df053d15c454f799d82a830ed2bdf1f (127.15.254.1:35729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9cb450e8229d46bcb8e1d6867d52e931: Network error: Client connection negotiation failed: client connection to 127.15.254.1:35729: connect: Connection refused (error 111)
I20260807 06:26:50.363410 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 11697
I20260807 06:26:50.376448 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 11201
2026-08-07T06:26:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2765 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-07T06:26:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:26:50Z Disabled control of system clock
I20260807 06:26:50.413058 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:42867
--webserver_interface=127.15.254.62
--webserver_port=0
--builtin_ntp_servers=127.15.254.20:41007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:42867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 06:26:50.528708 11939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:50.528980 11939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:50.529017 11939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:50.533159 11939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:26:50.533218 11939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:50.533234 11939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:50.533250 11939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:26:50.533267 11939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:26:50.538100 11939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:41007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:42867
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:42867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:50.539279 11939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:50.540872 11939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:50.549614 11949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:50.549607 11948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:50.549608 11953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:50.549671 11939 server_base.cc:1034] running on GCE node
I20260807 06:26:50.550382 11939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:50.551479 11939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:50.552665 11939 hybrid_clock.cc:648] HybridClock initialized: now 1786084010552639 us; error 45 us; skew 500 ppm
I20260807 06:26:50.559527 11939 webserver.cc:460] Webserver started at http://127.15.254.62:41585/ using document root <none> and password file <none>
I20260807 06:26:50.560163 11939 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:50.560258 11939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:50.560559 11939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:50.562707 11939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "e94a63b40cfc4d33ad1365330a7f1fa9"
format_stamp: "Formatted at 2026-08-07 06:26:50 on dist-test-slave-44d2"
server_key: "6fe303756a01d2de0d2324483e3360d9"
server_key_iv: "c9fa945beb6b56be2d35529afdf8f77c"
server_key_version: "encryptionkey@0"
I20260807 06:26:50.563275 11939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "e94a63b40cfc4d33ad1365330a7f1fa9"
format_stamp: "Formatted at 2026-08-07 06:26:50 on dist-test-slave-44d2"
server_key: "6fe303756a01d2de0d2324483e3360d9"
server_key_iv: "c9fa945beb6b56be2d35529afdf8f77c"
server_key_version: "encryptionkey@0"
I20260807 06:26:50.567067 11939 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260807 06:26:50.569597 11964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:50.570892 11939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 06:26:50.571039 11939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "e94a63b40cfc4d33ad1365330a7f1fa9"
format_stamp: "Formatted at 2026-08-07 06:26:50 on dist-test-slave-44d2"
server_key: "6fe303756a01d2de0d2324483e3360d9"
server_key_iv: "c9fa945beb6b56be2d35529afdf8f77c"
server_key_version: "encryptionkey@0"
I20260807 06:26:50.571148 11939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:50.589282 11939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:50.589982 11939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:50.590178 11939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:50.597399 11939 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:42867
I20260807 06:26:50.597407 12031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:42867 every 8 connection(s)
I20260807 06:26:50.598407 11939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:26:50.599290 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 11939
I20260807 06:26:50.599434 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:26:50.599725 16376 external_mini_cluster.cc:1428] Setting key 45c9295f402bf8f427090e6214194af3
I20260807 06:26:50.602108 12032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:50.608842 12032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e94a63b40cfc4d33ad1365330a7f1fa9: Bootstrap starting.
I20260807 06:26:50.609426 12035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:50.601204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50398 (local address 127.15.254.62:42867)
0807 06:26:50.601818 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:50.601834 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:50.601869 (+ 35us) server_negotiation.cc:407] Connection header received
0807 06:26:50.602613 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:50.602632 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:50.603032 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:50.603336 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:50.603672 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.604714 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.605959 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.606428 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.606567 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:50.607464 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:50.607469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:50.607482 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:26:50.607889 (+ 407us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:50.607950 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:26:50.607972 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:50.608055 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:50.608230 (+ 175us) server_negotiation.cc:299] Negotiation successful
0807 06:26:50.608588 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":138,"threads_started":1}
I20260807 06:26:50.611336 12032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e94a63b40cfc4d33ad1365330a7f1fa9: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:50.612221 12032 log.cc:824] T 00000000000000000000000000000000 P e94a63b40cfc4d33ad1365330a7f1fa9: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:50.614251 12032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e94a63b40cfc4d33ad1365330a7f1fa9: No bootstrap required, opened a new log
I20260807 06:26:50.617933 12032 raft_consensus.cc:348] T 00000000000000000000000000000000 P e94a63b40cfc4d33ad1365330a7f1fa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94a63b40cfc4d33ad1365330a7f1fa9" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 42867 } }
I20260807 06:26:50.618162 12032 raft_consensus.cc:374] T 00000000000000000000000000000000 P e94a63b40cfc4d33ad1365330a7f1fa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:50.618198 12032 raft_consensus.cc:729] T 00000000000000000000000000000000 P e94a63b40cfc4d33ad1365330a7f1fa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94a63b40cfc4d33ad1365330a7f1fa9, State: Initialized, Role: FOLLOWER
I20260807 06:26:50.618556 12032 consensus_queue.cc:260] T 00000000000000000000000000000000 P e94a63b40cfc4d33ad1365330a7f1fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94a63b40cfc4d33ad1365330a7f1fa9" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 42867 } }
I20260807 06:26:50.618664 12032 raft_consensus.cc:388] T 00000000000000000000000000000000 P e94a63b40cfc4d33ad1365330a7f1fa9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:26:50.618709 12032 raft_consensus.cc:482] T 00000000000000000000000000000000 P e94a63b40cfc4d33ad1365330a7f1fa9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:26:50.618777 12032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e94a63b40cfc4d33ad1365330a7f1fa9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:50.619580 12032 raft_consensus.cc:504] T 00000000000000000000000000000000 P e94a63b40cfc4d33ad1365330a7f1fa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94a63b40cfc4d33ad1365330a7f1fa9" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 42867 } }
I20260807 06:26:50.619866 12032 leader_election.cc:302] T 00000000000000000000000000000000 P e94a63b40cfc4d33ad1365330a7f1fa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e94a63b40cfc4d33ad1365330a7f1fa9; no voters:
I20260807 06:26:50.620110 12032 leader_election.cc:288] T 00000000000000000000000000000000 P e94a63b40cfc4d33ad1365330a7f1fa9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:26:50.620241 12041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e94a63b40cfc4d33ad1365330a7f1fa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:26:50.620537 12041 raft_consensus.cc:686] T 00000000000000000000000000000000 P e94a63b40cfc4d33ad1365330a7f1fa9 [term 1 LEADER]: Becoming Leader. State: Replica: e94a63b40cfc4d33ad1365330a7f1fa9, State: Running, Role: LEADER
I20260807 06:26:50.620829 12041 consensus_queue.cc:237] T 00000000000000000000000000000000 P e94a63b40cfc4d33ad1365330a7f1fa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94a63b40cfc4d33ad1365330a7f1fa9" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 42867 } }
I20260807 06:26:50.621304 12032 sys_catalog.cc:564] T 00000000000000000000000000000000 P e94a63b40cfc4d33ad1365330a7f1fa9 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:26:50.622413 12044 sys_catalog.cc:455] T 00000000000000000000000000000000 P e94a63b40cfc4d33ad1365330a7f1fa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e94a63b40cfc4d33ad1365330a7f1fa9. Latest consensus state: current_term: 1 leader_uuid: "e94a63b40cfc4d33ad1365330a7f1fa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94a63b40cfc4d33ad1365330a7f1fa9" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 42867 } } }
I20260807 06:26:50.622541 12044 sys_catalog.cc:458] T 00000000000000000000000000000000 P e94a63b40cfc4d33ad1365330a7f1fa9 [sys.catalog]: This master's current role is: LEADER
I20260807 06:26:50.622852 12042 sys_catalog.cc:455] T 00000000000000000000000000000000 P e94a63b40cfc4d33ad1365330a7f1fa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e94a63b40cfc4d33ad1365330a7f1fa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94a63b40cfc4d33ad1365330a7f1fa9" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 42867 } } }
I20260807 06:26:50.622990 12042 sys_catalog.cc:458] T 00000000000000000000000000000000 P e94a63b40cfc4d33ad1365330a7f1fa9 [sys.catalog]: This master's current role is: LEADER
I20260807 06:26:50.623018 12057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:26:50.625996 12057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:26:50.631354 12057 catalog_manager.cc:1347] Generated new cluster ID: 0110cf4cbf6945d7884acc92361c7dbc
I20260807 06:26:50.631418 12057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:26:50.647063 12057 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:26:50.648542 12057 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:26:50.654630 12057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e94a63b40cfc4d33ad1365330a7f1fa9: Generated new TSK 0
I20260807 06:26:50.655294 12057 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:26:50.661454 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.1:0
--local_ip_for_outbound_sockets=127.15.254.1
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:42867
--builtin_ntp_servers=127.15.254.20:41007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 06:26:50.780036 12068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:50.780303 12068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:50.780342 12068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:50.784436 12068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:50.784546 12068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:50.784631 12068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.1
I20260807 06:26:50.789538 12068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:41007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.254.1
--webserver_port=0
--tserver_master_addrs=127.15.254.62:42867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.1
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:50.790704 12068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:50.792198 12068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:50.802973 12075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:50.802953 12078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:50.802976 12076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:50.803956 12068 server_base.cc:1034] running on GCE node
I20260807 06:26:50.804451 12068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:50.805274 12068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:50.806440 12068 hybrid_clock.cc:648] HybridClock initialized: now 1786084010806402 us; error 51 us; skew 500 ppm
I20260807 06:26:50.808660 12068 webserver.cc:460] Webserver started at http://127.15.254.1:36501/ using document root <none> and password file <none>
I20260807 06:26:50.809427 12068 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:50.809506 12068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:50.809737 12068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:50.811944 12068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-0/data/instance:
uuid: "5469444bc2954c3c94429286108561fc"
format_stamp: "Formatted at 2026-08-07 06:26:50 on dist-test-slave-44d2"
server_key: "d75e5c6c28f925c4af66c135c3f1027b"
server_key_iv: "c86837589693287ef6454a33b444c692"
server_key_version: "encryptionkey@0"
I20260807 06:26:50.812625 12068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance:
uuid: "5469444bc2954c3c94429286108561fc"
format_stamp: "Formatted at 2026-08-07 06:26:50 on dist-test-slave-44d2"
server_key: "d75e5c6c28f925c4af66c135c3f1027b"
server_key_iv: "c86837589693287ef6454a33b444c692"
server_key_version: "encryptionkey@0"
I20260807 06:26:50.817116 12068 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260807 06:26:50.819830 12086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:50.821235 12068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 06:26:50.821386 12068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-0/wal
uuid: "5469444bc2954c3c94429286108561fc"
format_stamp: "Formatted at 2026-08-07 06:26:50 on dist-test-slave-44d2"
server_key: "d75e5c6c28f925c4af66c135c3f1027b"
server_key_iv: "c86837589693287ef6454a33b444c692"
server_key_version: "encryptionkey@0"
I20260807 06:26:50.821511 12068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:50.837333 12068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:50.838116 12068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:50.838325 12068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:50.839155 12068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:26:50.840597 12068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:26:50.840669 12068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:50.840746 12068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:26:50.840785 12068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:50.850490 12068 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.1:35049
I20260807 06:26:50.850514 12251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.1:35049 every 8 connection(s)
I20260807 06:26:50.851572 12068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-0/data/info.pb
I20260807 06:26:50.859539 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 12068
I20260807 06:26:50.859706 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-0/wal/instance
I20260807 06:26:50.860141 16376 external_mini_cluster.cc:1428] Setting key fd74764602d30fee854ceb1fe9db2851
I20260807 06:26:50.861842 12035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:50.853599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:33653 (local address 127.15.254.62:42867)
0807 06:26:50.853757 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:50.853762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:50.855004 (+ 1242us) server_negotiation.cc:407] Connection header received
0807 06:26:50.856416 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:50.856419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:50.856476 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:50.856622 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:50.857597 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.858189 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.859451 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.859745 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.859858 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:50.861309 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:50.861313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:50.861315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:26:50.861326 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:50.861347 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:26:50.861355 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:50.861418 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:50.861614 (+ 196us) server_negotiation.cc:299] Negotiation successful
0807 06:26:50.861700 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 06:26:50.862507 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.2:0
--local_ip_for_outbound_sockets=127.15.254.2
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:42867
--builtin_ntp_servers=127.15.254.20:41007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 06:26:50.862792 12255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:50.854107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:42867 (local address 127.15.254.1:33653)
0807 06:26:50.854845 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:50.854889 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:50.856188 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:50.856826 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:50.856833 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:50.856920 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:26:50.857431 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:50.857444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.858318 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.858321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:50.859307 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:50.859321 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:50.859919 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:50.859924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:50.860162 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:50.860979 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:26:50.860982 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:26:50.861078 (+ 96us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:26:50.861161 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:26:50.861441 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:26:50.861451 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:26:50.861456 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 06:26:50.861601 (+ 145us) client_negotiation.cc:240] Negotiation successful
0807 06:26:50.861894 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":138,"threads_started":1}
I20260807 06:26:50.864022 12253 heartbeater.cc:347] Connected to a master server at 127.15.254.62:42867
I20260807 06:26:50.864291 12253 heartbeater.cc:464] Registering TS with master...
I20260807 06:26:50.864965 12253 heartbeater.cc:511] Master 127.15.254.62:42867 requested a full tablet report, sending...
I20260807 06:26:50.866688 11988 ts_manager.cc:194] Registered new tserver with Master: 5469444bc2954c3c94429286108561fc (127.15.254.1:35049)
I20260807 06:26:50.868359 11988 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.254.1:33653
W20260807 06:26:50.985602 12256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:50.985893 12256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:50.985931 12256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:50.990134 12256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:50.990204 12256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:50.990300 12256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.2
I20260807 06:26:50.995199 12256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:41007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.254.2
--webserver_port=0
--tserver_master_addrs=127.15.254.62:42867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.2
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:50.996402 12256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:50.997941 12256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:51.009106 12265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:51.009110 12261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:51.009106 12262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:51.009773 12256 server_base.cc:1034] running on GCE node
I20260807 06:26:51.010317 12256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:51.011127 12256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:51.012327 12256 hybrid_clock.cc:648] HybridClock initialized: now 1786084011012284 us; error 59 us; skew 500 ppm
I20260807 06:26:51.014582 12256 webserver.cc:460] Webserver started at http://127.15.254.2:35861/ using document root <none> and password file <none>
I20260807 06:26:51.015236 12256 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:51.015326 12256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:51.015604 12256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:51.017922 12256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-1/data/instance:
uuid: "ccc5039faea74d6c89665bdf217e2d27"
format_stamp: "Formatted at 2026-08-07 06:26:51 on dist-test-slave-44d2"
server_key: "1008dcf28b5284e8e94353a3fd57f2a6"
server_key_iv: "d7c3bbcc695ad4189b7ea9237426cfaa"
server_key_version: "encryptionkey@0"
I20260807 06:26:51.018540 12256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance:
uuid: "ccc5039faea74d6c89665bdf217e2d27"
format_stamp: "Formatted at 2026-08-07 06:26:51 on dist-test-slave-44d2"
server_key: "1008dcf28b5284e8e94353a3fd57f2a6"
server_key_iv: "d7c3bbcc695ad4189b7ea9237426cfaa"
server_key_version: "encryptionkey@0"
I20260807 06:26:51.022827 12256 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:26:51.025645 12275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:51.026979 12256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:26:51.027108 12256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-1/wal
uuid: "ccc5039faea74d6c89665bdf217e2d27"
format_stamp: "Formatted at 2026-08-07 06:26:51 on dist-test-slave-44d2"
server_key: "1008dcf28b5284e8e94353a3fd57f2a6"
server_key_iv: "d7c3bbcc695ad4189b7ea9237426cfaa"
server_key_version: "encryptionkey@0"
I20260807 06:26:51.027230 12256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:51.037657 12256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:51.038424 12256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:51.038640 12256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:51.039288 12256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:26:51.040315 12256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:26:51.040369 12256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:51.040448 12256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:26:51.040519 12256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:51.050082 12256 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.2:32871
I20260807 06:26:51.050124 12424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.2:32871 every 8 connection(s)
I20260807 06:26:51.051031 12256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-1/data/info.pb
I20260807 06:26:51.052830 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 12256
I20260807 06:26:51.053020 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-1/wal/instance
I20260807 06:26:51.053409 16376 external_mini_cluster.cc:1428] Setting key 3a22f6d8a178aec2c3697989d77dd88c
I20260807 06:26:51.055755 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.254.3:0
--local_ip_for_outbound_sockets=127.15.254.3
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:42867
--builtin_ntp_servers=127.15.254.20:41007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 06:26:51.062461 12035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:51.053200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:33483 (local address 127.15.254.62:42867)
0807 06:26:51.053378 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:51.053383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:51.054471 (+ 1088us) server_negotiation.cc:407] Connection header received
0807 06:26:51.056407 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:51.056410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:51.056459 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:51.056586 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:51.057895 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.058472 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.060057 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.060276 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.060363 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:51.061911 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:51.061914 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:51.061916 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:26:51.061931 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:51.061950 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:26:51.062004 (+ 54us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:51.062067 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:51.062283 (+ 216us) server_negotiation.cc:299] Negotiation successful
0807 06:26:51.062341 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 06:26:51.064023 12427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:51.053490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:42867 (local address 127.15.254.2:33483)
0807 06:26:51.054316 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:51.054371 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:51.056168 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:51.056781 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:51.056794 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:51.056920 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:26:51.057679 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:51.057703 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.058622 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.058625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:51.059918 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:51.059934 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.060451 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.060459 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:51.060734 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:51.061553 (+ 819us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:26:51.061558 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:26:51.061623 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:26:51.061726 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:26:51.062099 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:26:51.062112 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:26:51.062122 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 06:26:51.062284 (+ 162us) client_negotiation.cc:240] Negotiation successful
0807 06:26:51.062639 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":128,"threads_started":1}
I20260807 06:26:51.065621 12425 heartbeater.cc:347] Connected to a master server at 127.15.254.62:42867
I20260807 06:26:51.065904 12425 heartbeater.cc:464] Registering TS with master...
I20260807 06:26:51.066505 12425 heartbeater.cc:511] Master 127.15.254.62:42867 requested a full tablet report, sending...
I20260807 06:26:51.067684 11988 ts_manager.cc:194] Registered new tserver with Master: ccc5039faea74d6c89665bdf217e2d27 (127.15.254.2:32871)
I20260807 06:26:51.068336 11988 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.254.2:33483
W20260807 06:26:51.177438 12428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:51.177728 12428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:51.177798 12428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:51.181794 12428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:51.181866 12428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:51.181954 12428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.254.3
I20260807 06:26:51.186682 12428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:41007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.254.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.254.3
--webserver_port=0
--tserver_master_addrs=127.15.254.62:42867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.254.3
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:51.187816 12428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:51.189273 12428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:51.199546 12436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:51.199633 12428 server_base.cc:1034] running on GCE node
W20260807 06:26:51.199558 12437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:51.199574 12440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:51.200181 12428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:51.200955 12428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:51.202149 12428 hybrid_clock.cc:648] HybridClock initialized: now 1786084011202129 us; error 34 us; skew 500 ppm
I20260807 06:26:51.208073 12428 webserver.cc:460] Webserver started at http://127.15.254.3:42433/ using document root <none> and password file <none>
I20260807 06:26:51.208915 12428 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:51.209029 12428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:51.209309 12428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:51.211915 12428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-2/data/instance:
uuid: "e12d0cb79b834e36b71d745ca40bede7"
format_stamp: "Formatted at 2026-08-07 06:26:51 on dist-test-slave-44d2"
server_key: "3562fe8b1cf08e56ed4ec047e9ef8937"
server_key_iv: "f24fb5012fc57b236c8ba5b314002918"
server_key_version: "encryptionkey@0"
I20260807 06:26:51.212684 12428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance:
uuid: "e12d0cb79b834e36b71d745ca40bede7"
format_stamp: "Formatted at 2026-08-07 06:26:51 on dist-test-slave-44d2"
server_key: "3562fe8b1cf08e56ed4ec047e9ef8937"
server_key_iv: "f24fb5012fc57b236c8ba5b314002918"
server_key_version: "encryptionkey@0"
I20260807 06:26:51.217281 12428 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:26:51.220074 12447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:51.221517 12428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:26:51.221663 12428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-2/wal
uuid: "e12d0cb79b834e36b71d745ca40bede7"
format_stamp: "Formatted at 2026-08-07 06:26:51 on dist-test-slave-44d2"
server_key: "3562fe8b1cf08e56ed4ec047e9ef8937"
server_key_iv: "f24fb5012fc57b236c8ba5b314002918"
server_key_version: "encryptionkey@0"
I20260807 06:26:51.221900 12428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:51.240764 12428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:51.241529 12428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:51.241742 12428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:51.242380 12428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:26:51.243399 12428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:26:51.243469 12428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:51.243568 12428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:26:51.243614 12428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:51.253278 12428 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.3:34421
I20260807 06:26:51.253310 12604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.3:34421 every 8 connection(s)
I20260807 06:26:51.254231 12428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-2/data/info.pb
I20260807 06:26:51.254760 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 12428
I20260807 06:26:51.254899 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-2/wal/instance
I20260807 06:26:51.255273 16376 external_mini_cluster.cc:1428] Setting key 1f48d4a136daa47cc764ea6dc3c5a31d
I20260807 06:26:51.264533 12035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:51.256300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.3:53263 (local address 127.15.254.62:42867)
0807 06:26:51.256445 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:51.256449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:51.257996 (+ 1547us) server_negotiation.cc:407] Connection header received
0807 06:26:51.259553 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:51.259556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:51.259613 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:51.259692 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:51.260665 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.261209 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.262388 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.262627 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.262732 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:51.263908 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:51.263911 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:51.263913 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:26:51.263927 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:51.263948 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:26:51.263957 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:51.264018 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:51.264262 (+ 244us) server_negotiation.cc:299] Negotiation successful
0807 06:26:51.264332 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 06:26:51.265389 12609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:51.256933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:42867 (local address 127.15.254.3:53263)
0807 06:26:51.257807 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:51.257880 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:51.259336 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:51.259868 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:51.259875 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:51.259966 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:26:51.260473 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:51.260542 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.261340 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.261344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:51.262259 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:51.262269 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.262792 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.262796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:51.262981 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:51.263631 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:26:51.263633 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:26:51.263675 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:26:51.263743 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:26:51.264081 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:26:51.264090 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:26:51.264095 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 06:26:51.264226 (+ 131us) client_negotiation.cc:240] Negotiation successful
0807 06:26:51.264561 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":180,"threads_started":1}
I20260807 06:26:51.266570 12606 heartbeater.cc:347] Connected to a master server at 127.15.254.62:42867
I20260807 06:26:51.266834 12606 heartbeater.cc:464] Registering TS with master...
I20260807 06:26:51.267457 12606 heartbeater.cc:511] Master 127.15.254.62:42867 requested a full tablet report, sending...
I20260807 06:26:51.268841 11988 ts_manager.cc:194] Registered new tserver with Master: e12d0cb79b834e36b71d745ca40bede7 (127.15.254.3:34421)
I20260807 06:26:51.269721 11988 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.254.3:53263
I20260807 06:26:51.270211 16376 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:26:51.276722 12035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:51.272619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50406 (local address 127.15.254.62:42867)
0807 06:26:51.272765 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:51.272771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:51.272973 (+ 202us) server_negotiation.cc:407] Connection header received
0807 06:26:51.273141 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:51.273144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:51.273193 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:51.273276 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:51.273560 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.274132 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.275039 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.275267 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.275376 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:51.276258 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:51.276262 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:51.276264 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:26:51.276283 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:51.276302 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:26:51.276310 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:51.276390 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:51.276514 (+ 124us) server_negotiation.cc:299] Negotiation successful
0807 06:26:51.276574 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260807 06:26:51.279913 11988 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:26:51.282965 11988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:26:51.298604 12623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:51.292228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:32871 (local address 127.0.0.1:45566)
0807 06:26:51.293523 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:51.293575 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:51.293706 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:51.294490 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:51.294493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:51.294521 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:51.294704 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:51.294711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.295747 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.295752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:51.297084 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:51.297093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.298018 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.298022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:51.298135 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:51.298181 (+ 46us) client_negotiation.cc:769] Sending connection context
0807 06:26:51.298271 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 06:26:51.298398 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":109,"spinlock_wait_cycles":701568,"thread_start_us":84,"threads_started":1}
I20260807 06:26:51.299270 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:51.293158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:35049 (local address 127.0.0.1:43416)
0807 06:26:51.293521 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:51.293574 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:51.293706 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:51.294303 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:51.294309 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:51.294333 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:51.294506 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:51.294514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.296035 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.296038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:51.297523 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:51.297535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.298811 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.298814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:51.298921 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:51.298957 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 06:26:51.299013 (+ 56us) client_negotiation.cc:240] Negotiation successful
0807 06:26:51.299088 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":121,"spinlock_wait_cycles":38784,"thread_start_us":112,"threads_started":1}
I20260807 06:26:51.299640 12627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:51.293045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.15.254.2:32871)
0807 06:26:51.294026 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:51.294035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:51.294056 (+ 21us) server_negotiation.cc:407] Connection header received
0807 06:26:51.294118 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:51.294122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:51.294304 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:51.294499 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:51.294834 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.295625 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.297207 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.297880 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.298810 (+ 930us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:51.299012 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:51.299064 (+ 52us) server_negotiation.cc:299] Negotiation successful
0807 06:26:51.299169 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":866,"thread_start_us":158,"threads_started":1}
I20260807 06:26:51.299952 12625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:51.292289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46884 (local address 127.15.254.3:34421)
0807 06:26:51.293238 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:51.293256 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:51.293678 (+ 422us) server_negotiation.cc:407] Connection header received
0807 06:26:51.293905 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:51.293912 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:51.294131 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:51.294267 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:51.294926 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.296050 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.297662 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.298496 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.298601 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:51.298820 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:51.299643 (+ 823us) server_negotiation.cc:299] Negotiation successful
0807 06:26:51.299761 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":826,"thread_start_us":205,"threads_started":1}
I20260807 06:26:51.300508 12626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:51.292780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43416 (local address 127.15.254.1:35049)
0807 06:26:51.293848 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:51.293864 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:51.293888 (+ 24us) server_negotiation.cc:407] Connection header received
0807 06:26:51.293960 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:51.293964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:51.294130 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:51.294279 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:51.294619 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.295870 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.297700 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.298652 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.299924 (+ 1272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:51.300148 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:51.300213 (+ 65us) server_negotiation.cc:299] Negotiation successful
0807 06:26:51.300314 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":136,"threads_started":1}
I20260807 06:26:51.300822 12624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:51.292082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:34421 (local address 127.0.0.1:46884)
0807 06:26:51.293523 (+ 1441us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:51.293575 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:51.293745 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:51.294494 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:51.294497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:51.294522 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:51.294736 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:51.294741 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.296172 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.296175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:51.297523 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:51.297535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.299164 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.299167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:51.299263 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:51.299286 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 06:26:51.300397 (+ 1111us) client_negotiation.cc:240] Negotiation successful
0807 06:26:51.300502 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":213,"spinlock_wait_cycles":18560,"thread_start_us":105,"threads_started":1}
I20260807 06:26:51.304170 12510 tablet_service.cc:1467] Processing CreateTablet for tablet 890e02b1642849678939b558a26ae07e (DEFAULT_TABLE table=test-table [id=2c405db6348244f0ad0de0e13aed1f45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:26:51.304169 12158 tablet_service.cc:1467] Processing CreateTablet for tablet 890e02b1642849678939b558a26ae07e (DEFAULT_TABLE table=test-table [id=2c405db6348244f0ad0de0e13aed1f45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:26:51.305423 12158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 890e02b1642849678939b558a26ae07e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:51.305423 12510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 890e02b1642849678939b558a26ae07e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:51.306466 12336 tablet_service.cc:1467] Processing CreateTablet for tablet 890e02b1642849678939b558a26ae07e (DEFAULT_TABLE table=test-table [id=2c405db6348244f0ad0de0e13aed1f45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:26:51.307742 12336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 890e02b1642849678939b558a26ae07e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:51.314692 12632 tablet_bootstrap.cc:492] T 890e02b1642849678939b558a26ae07e P e12d0cb79b834e36b71d745ca40bede7: Bootstrap starting.
I20260807 06:26:51.317411 12632 tablet_bootstrap.cc:654] T 890e02b1642849678939b558a26ae07e P e12d0cb79b834e36b71d745ca40bede7: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:51.317652 12629 tablet_bootstrap.cc:492] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc: Bootstrap starting.
I20260807 06:26:51.318405 12632 log.cc:824] T 890e02b1642849678939b558a26ae07e P e12d0cb79b834e36b71d745ca40bede7: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:51.319281 12634 tablet_bootstrap.cc:492] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27: Bootstrap starting.
I20260807 06:26:51.320215 12629 tablet_bootstrap.cc:654] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:51.321134 12632 tablet_bootstrap.cc:492] T 890e02b1642849678939b558a26ae07e P e12d0cb79b834e36b71d745ca40bede7: No bootstrap required, opened a new log
I20260807 06:26:51.321354 12632 ts_tablet_manager.cc:1397] T 890e02b1642849678939b558a26ae07e P e12d0cb79b834e36b71d745ca40bede7: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260807 06:26:51.321319 12629 log.cc:824] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:51.323889 12634 tablet_bootstrap.cc:654] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:51.324826 12629 tablet_bootstrap.cc:492] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc: No bootstrap required, opened a new log
I20260807 06:26:51.324846 12634 log.cc:824] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:51.325178 12629 ts_tablet_manager.cc:1397] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260807 06:26:51.326258 12632 raft_consensus.cc:348] T 890e02b1642849678939b558a26ae07e P e12d0cb79b834e36b71d745ca40bede7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12d0cb79b834e36b71d745ca40bede7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34421 } } peers { permanent_uuid: "ccc5039faea74d6c89665bdf217e2d27" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 32871 } } peers { permanent_uuid: "5469444bc2954c3c94429286108561fc" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35049 } }
I20260807 06:26:51.326612 12632 raft_consensus.cc:374] T 890e02b1642849678939b558a26ae07e P e12d0cb79b834e36b71d745ca40bede7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:51.326689 12632 raft_consensus.cc:729] T 890e02b1642849678939b558a26ae07e P e12d0cb79b834e36b71d745ca40bede7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e12d0cb79b834e36b71d745ca40bede7, State: Initialized, Role: FOLLOWER
I20260807 06:26:51.327062 12634 tablet_bootstrap.cc:492] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27: No bootstrap required, opened a new log
I20260807 06:26:51.327252 12634 ts_tablet_manager.cc:1397] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260807 06:26:51.327296 12632 consensus_queue.cc:260] T 890e02b1642849678939b558a26ae07e P e12d0cb79b834e36b71d745ca40bede7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12d0cb79b834e36b71d745ca40bede7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34421 } } peers { permanent_uuid: "ccc5039faea74d6c89665bdf217e2d27" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 32871 } } peers { permanent_uuid: "5469444bc2954c3c94429286108561fc" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35049 } }
I20260807 06:26:51.328454 12632 ts_tablet_manager.cc:1428] T 890e02b1642849678939b558a26ae07e P e12d0cb79b834e36b71d745ca40bede7: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260807 06:26:51.328878 12606 heartbeater.cc:503] Master 127.15.254.62:42867 was elected leader, sending a full tablet report...
I20260807 06:26:51.332300 12629 raft_consensus.cc:348] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12d0cb79b834e36b71d745ca40bede7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34421 } } peers { permanent_uuid: "ccc5039faea74d6c89665bdf217e2d27" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 32871 } } peers { permanent_uuid: "5469444bc2954c3c94429286108561fc" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35049 } }
I20260807 06:26:51.332417 12634 raft_consensus.cc:348] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12d0cb79b834e36b71d745ca40bede7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34421 } } peers { permanent_uuid: "ccc5039faea74d6c89665bdf217e2d27" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 32871 } } peers { permanent_uuid: "5469444bc2954c3c94429286108561fc" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35049 } }
I20260807 06:26:51.332758 12634 raft_consensus.cc:374] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:51.332705 12629 raft_consensus.cc:374] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:51.332825 12634 raft_consensus.cc:729] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccc5039faea74d6c89665bdf217e2d27, State: Initialized, Role: FOLLOWER
I20260807 06:26:51.332934 12629 raft_consensus.cc:729] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5469444bc2954c3c94429286108561fc, State: Initialized, Role: FOLLOWER
I20260807 06:26:51.333400 12634 consensus_queue.cc:260] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12d0cb79b834e36b71d745ca40bede7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34421 } } peers { permanent_uuid: "ccc5039faea74d6c89665bdf217e2d27" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 32871 } } peers { permanent_uuid: "5469444bc2954c3c94429286108561fc" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35049 } }
I20260807 06:26:51.333451 12629 consensus_queue.cc:260] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12d0cb79b834e36b71d745ca40bede7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34421 } } peers { permanent_uuid: "ccc5039faea74d6c89665bdf217e2d27" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 32871 } } peers { permanent_uuid: "5469444bc2954c3c94429286108561fc" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35049 } }
I20260807 06:26:51.334188 12253 heartbeater.cc:503] Master 127.15.254.62:42867 was elected leader, sending a full tablet report...
I20260807 06:26:51.334525 12634 ts_tablet_manager.cc:1428] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260807 06:26:51.334570 12629 ts_tablet_manager.cc:1428] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260807 06:26:51.335125 12425 heartbeater.cc:503] Master 127.15.254.62:42867 was elected leader, sending a full tablet report...
W20260807 06:26:51.352895 12254 tablet.cc:2369] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 06:26:51.505585 12608 tablet.cc:2369] T 890e02b1642849678939b558a26ae07e P e12d0cb79b834e36b71d745ca40bede7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 06:26:51.561106 12426 tablet.cc:2369] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:26:51.626109 12638 raft_consensus.cc:482] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:26:51.626250 12640 raft_consensus.cc:482] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:26:51.626420 12638 raft_consensus.cc:504] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12d0cb79b834e36b71d745ca40bede7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34421 } } peers { permanent_uuid: "ccc5039faea74d6c89665bdf217e2d27" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 32871 } } peers { permanent_uuid: "5469444bc2954c3c94429286108561fc" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35049 } }
I20260807 06:26:51.626420 12640 raft_consensus.cc:504] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12d0cb79b834e36b71d745ca40bede7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34421 } } peers { permanent_uuid: "ccc5039faea74d6c89665bdf217e2d27" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 32871 } } peers { permanent_uuid: "5469444bc2954c3c94429286108561fc" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35049 } }
I20260807 06:26:51.627876 12638 leader_election.cc:288] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e12d0cb79b834e36b71d745ca40bede7 (127.15.254.3:34421), 5469444bc2954c3c94429286108561fc (127.15.254.1:35049)
I20260807 06:26:51.628432 12640 leader_election.cc:288] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e12d0cb79b834e36b71d745ca40bede7 (127.15.254.3:34421), ccc5039faea74d6c89665bdf217e2d27 (127.15.254.2:32871)
I20260807 06:26:51.633256 12625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:51.628314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:46229 (local address 127.15.254.3:34421)
0807 06:26:51.628665 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:51.628671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:51.628692 (+ 21us) server_negotiation.cc:407] Connection header received
0807 06:26:51.628759 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:51.628763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:51.628823 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:51.628911 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:51.629349 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.630133 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.631857 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.632533 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.632660 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:51.632712 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:51.633036 (+ 324us) server_negotiation.cc:299] Negotiation successful
0807 06:26:51.633107 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":268}
I20260807 06:26:51.633718 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:51.628155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:34421 (local address 127.15.254.2:46229)
0807 06:26:51.628545 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:51.628560 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:51.628662 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:51.629011 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:51.629015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:51.629041 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:51.629177 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:51.629185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.630281 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.630287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:51.631709 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:51.631720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.632802 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.632805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:51.632892 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:51.632916 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 06:26:51.633467 (+ 551us) client_negotiation.cc:240] Negotiation successful
0807 06:26:51.633544 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":23,"thread_start_us":92,"threads_started":1}
I20260807 06:26:51.633831 12627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:51.629478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:33291 (local address 127.15.254.2:32871)
0807 06:26:51.629711 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:51.629716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:51.629728 (+ 12us) server_negotiation.cc:407] Connection header received
0807 06:26:51.629772 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:51.629778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:51.629826 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:51.629893 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:51.630279 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.631077 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.632619 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.633167 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.633260 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:51.633310 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:51.633566 (+ 256us) server_negotiation.cc:299] Negotiation successful
0807 06:26:51.633655 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":164}
I20260807 06:26:51.634292 12626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:51.629349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.2:51849 (local address 127.15.254.1:35049)
0807 06:26:51.629615 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:51.629619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:51.629675 (+ 56us) server_negotiation.cc:407] Connection header received
0807 06:26:51.629844 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:51.629847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:51.629906 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:51.629951 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:51.630340 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.631158 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.632680 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.633275 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.633367 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:51.633418 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:51.633760 (+ 342us) server_negotiation.cc:299] Negotiation successful
0807 06:26:51.633842 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":201}
I20260807 06:26:51.634307 12645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:51.628378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.254.1:50243 (local address 127.15.254.3:34421)
0807 06:26:51.629217 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:51.629226 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:51.629239 (+ 13us) server_negotiation.cc:407] Connection header received
0807 06:26:51.629281 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:51.629284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:51.629333 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:51.629419 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:26:51.629838 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.630627 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.632176 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.632900 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.633005 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:51.633057 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:51.634109 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0807 06:26:51.634184 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":765,"thread_start_us":122,"threads_started":1}
I20260807 06:26:51.634325 12644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:51.628171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.2:32871 (local address 127.15.254.1:33291)
0807 06:26:51.628737 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:51.628751 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:51.628841 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:51.629998 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:51.630001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:51.630017 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:51.630140 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:51.630147 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.631254 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.631257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:51.632438 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:51.632449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.633324 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.633330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:51.633431 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:51.633450 (+ 19us) client_negotiation.cc:769] Sending connection context
0807 06:26:51.633743 (+ 293us) client_negotiation.cc:240] Negotiation successful
0807 06:26:51.634012 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":108,"threads_started":1}
I20260807 06:26:51.634383 12540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "890e02b1642849678939b558a26ae07e" candidate_uuid: "ccc5039faea74d6c89665bdf217e2d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e12d0cb79b834e36b71d745ca40bede7" is_pre_election: true
I20260807 06:26:51.634704 12647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:51.629251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:35049 (local address 127.15.254.2:51849)
0807 06:26:51.629592 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:51.629607 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:51.629741 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:51.630068 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:51.630071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:51.630086 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:51.630207 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:51.630217 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.631332 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.631335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:51.632511 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:51.632523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.633512 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.633515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:51.633613 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:51.633634 (+ 21us) client_negotiation.cc:769] Sending connection context
0807 06:26:51.633890 (+ 256us) client_negotiation.cc:240] Negotiation successful
0807 06:26:51.633966 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":69,"threads_started":1}
I20260807 06:26:51.634748 12540 raft_consensus.cc:2454] T 890e02b1642849678939b558a26ae07e P e12d0cb79b834e36b71d745ca40bede7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccc5039faea74d6c89665bdf217e2d27 in term 0.
I20260807 06:26:51.634907 12641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:51.628158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.3:34421 (local address 127.15.254.1:50243)
0807 06:26:51.628533 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:51.628550 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:26:51.628662 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:26:51.629548 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:26:51.629551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:26:51.629571 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:26:51.629703 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:51.629712 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.630751 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.630754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:51.632027 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:26:51.632038 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.633121 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.633126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:26:51.633540 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:51.633580 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 06:26:51.633644 (+ 64us) client_negotiation.cc:240] Negotiation successful
0807 06:26:51.633715 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":297,"thread_start_us":92,"threads_started":1}
I20260807 06:26:51.635052 12364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "890e02b1642849678939b558a26ae07e" candidate_uuid: "5469444bc2954c3c94429286108561fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccc5039faea74d6c89665bdf217e2d27" is_pre_election: true
I20260807 06:26:51.635251 12364 raft_consensus.cc:2454] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5469444bc2954c3c94429286108561fc in term 0.
I20260807 06:26:51.635259 12185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "890e02b1642849678939b558a26ae07e" candidate_uuid: "ccc5039faea74d6c89665bdf217e2d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5469444bc2954c3c94429286108561fc" is_pre_election: true
I20260807 06:26:51.635299 12280 leader_election.cc:302] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccc5039faea74d6c89665bdf217e2d27, e12d0cb79b834e36b71d745ca40bede7; no voters:
I20260807 06:26:51.635398 12540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "890e02b1642849678939b558a26ae07e" candidate_uuid: "5469444bc2954c3c94429286108561fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e12d0cb79b834e36b71d745ca40bede7" is_pre_election: true
I20260807 06:26:51.635430 12185 raft_consensus.cc:2454] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccc5039faea74d6c89665bdf217e2d27 in term 0.
I20260807 06:26:51.635511 12540 raft_consensus.cc:2454] T 890e02b1642849678939b558a26ae07e P e12d0cb79b834e36b71d745ca40bede7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5469444bc2954c3c94429286108561fc in term 0.
I20260807 06:26:51.635639 12638 raft_consensus.cc:2781] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:26:51.635735 12638 raft_consensus.cc:482] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:26:51.635793 12638 raft_consensus.cc:3037] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:51.635906 12090 leader_election.cc:302] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5469444bc2954c3c94429286108561fc, ccc5039faea74d6c89665bdf217e2d27; no voters:
I20260807 06:26:51.636224 12640 raft_consensus.cc:2781] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:26:51.636312 12640 raft_consensus.cc:482] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:26:51.636356 12640 raft_consensus.cc:3037] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:51.637070 12638 raft_consensus.cc:504] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12d0cb79b834e36b71d745ca40bede7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34421 } } peers { permanent_uuid: "ccc5039faea74d6c89665bdf217e2d27" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 32871 } } peers { permanent_uuid: "5469444bc2954c3c94429286108561fc" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35049 } }
I20260807 06:26:51.637463 12638 leader_election.cc:288] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27 [CANDIDATE]: Term 1 election: Requested vote from peers e12d0cb79b834e36b71d745ca40bede7 (127.15.254.3:34421), 5469444bc2954c3c94429286108561fc (127.15.254.1:35049)
I20260807 06:26:51.637535 12640 raft_consensus.cc:504] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12d0cb79b834e36b71d745ca40bede7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34421 } } peers { permanent_uuid: "ccc5039faea74d6c89665bdf217e2d27" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 32871 } } peers { permanent_uuid: "5469444bc2954c3c94429286108561fc" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35049 } }
I20260807 06:26:51.637800 12540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "890e02b1642849678939b558a26ae07e" candidate_uuid: "ccc5039faea74d6c89665bdf217e2d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e12d0cb79b834e36b71d745ca40bede7"
I20260807 06:26:51.637812 12185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "890e02b1642849678939b558a26ae07e" candidate_uuid: "ccc5039faea74d6c89665bdf217e2d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5469444bc2954c3c94429286108561fc"
I20260807 06:26:51.637923 12640 leader_election.cc:288] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc [CANDIDATE]: Term 1 election: Requested vote from peers e12d0cb79b834e36b71d745ca40bede7 (127.15.254.3:34421), ccc5039faea74d6c89665bdf217e2d27 (127.15.254.2:32871)
I20260807 06:26:51.637921 12540 raft_consensus.cc:3037] T 890e02b1642849678939b558a26ae07e P e12d0cb79b834e36b71d745ca40bede7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:51.638041 12185 raft_consensus.cc:2379] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ccc5039faea74d6c89665bdf217e2d27 in current term 1: Already voted for candidate 5469444bc2954c3c94429286108561fc in this term.
I20260807 06:26:51.638381 12539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "890e02b1642849678939b558a26ae07e" candidate_uuid: "5469444bc2954c3c94429286108561fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e12d0cb79b834e36b71d745ca40bede7"
I20260807 06:26:51.638381 12364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "890e02b1642849678939b558a26ae07e" candidate_uuid: "5469444bc2954c3c94429286108561fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccc5039faea74d6c89665bdf217e2d27"
I20260807 06:26:51.638554 12364 raft_consensus.cc:2379] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5469444bc2954c3c94429286108561fc in current term 1: Already voted for candidate ccc5039faea74d6c89665bdf217e2d27 in this term.
I20260807 06:26:51.639164 12090 leader_election.cc:302] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5469444bc2954c3c94429286108561fc; no voters: ccc5039faea74d6c89665bdf217e2d27, e12d0cb79b834e36b71d745ca40bede7
I20260807 06:26:51.639262 12540 raft_consensus.cc:2454] T 890e02b1642849678939b558a26ae07e P e12d0cb79b834e36b71d745ca40bede7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccc5039faea74d6c89665bdf217e2d27 in term 1.
I20260807 06:26:51.639441 12640 raft_consensus.cc:2726] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260807 06:26:51.639624 12280 leader_election.cc:302] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ccc5039faea74d6c89665bdf217e2d27, e12d0cb79b834e36b71d745ca40bede7; no voters: 5469444bc2954c3c94429286108561fc
I20260807 06:26:51.639825 12638 raft_consensus.cc:2781] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:26:51.640101 12638 raft_consensus.cc:686] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27 [term 1 LEADER]: Becoming Leader. State: Replica: ccc5039faea74d6c89665bdf217e2d27, State: Running, Role: LEADER
I20260807 06:26:51.640457 12638 consensus_queue.cc:237] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12d0cb79b834e36b71d745ca40bede7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34421 } } peers { permanent_uuid: "ccc5039faea74d6c89665bdf217e2d27" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 32871 } } peers { permanent_uuid: "5469444bc2954c3c94429286108561fc" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35049 } }
I20260807 06:26:51.643687 11988 catalog_manager.cc:5495] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccc5039faea74d6c89665bdf217e2d27 (127.15.254.2). New cstate: current_term: 1 leader_uuid: "ccc5039faea74d6c89665bdf217e2d27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12d0cb79b834e36b71d745ca40bede7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccc5039faea74d6c89665bdf217e2d27" member_type: VOTER last_known_addr { host: "127.15.254.2" port: 32871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5469444bc2954c3c94429286108561fc" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20260807 06:26:51.666417 12627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:51.662837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45578 (local address 127.15.254.2:32871)
0807 06:26:51.662978 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:51.662984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:51.663016 (+ 32us) server_negotiation.cc:407] Connection header received
0807 06:26:51.663101 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:51.663103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:51.663155 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:51.663263 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:26:51.663518 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.664052 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.664963 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:51.665189 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:51.665298 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:51.665595 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:26:51.666020 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:26:51.666150 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:51.666215 (+ 65us) server_negotiation.cc:299] Negotiation successful
0807 06:26:51.666270 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 06:26:51.672624 12540 raft_consensus.cc:1261] T 890e02b1642849678939b558a26ae07e P e12d0cb79b834e36b71d745ca40bede7 [term 1 FOLLOWER]: Refusing update from remote peer ccc5039faea74d6c89665bdf217e2d27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:26:51.672652 12185 raft_consensus.cc:1261] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc [term 1 FOLLOWER]: Refusing update from remote peer ccc5039faea74d6c89665bdf217e2d27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:26:51.673230 12638 consensus_queue.cc:1035] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5469444bc2954c3c94429286108561fc" member_type: VOTER last_known_addr { host: "127.15.254.1" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:26:51.673489 12648 consensus_queue.cc:1035] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e12d0cb79b834e36b71d745ca40bede7" member_type: VOTER last_known_addr { host: "127.15.254.3" port: 34421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:26:51.681401 12653 mvcc.cc:205] Tried to move back new op lower bound from 7315800111803613184 to 7315800111681888256. Current Snapshot: MvccSnapshot[applied={T|T < 7315800111803613184}]
I20260807 06:26:51.684847 12657 mvcc.cc:205] Tried to move back new op lower bound from 7315800111803613184 to 7315800111681888256. Current Snapshot: MvccSnapshot[applied={T|T < 7315800111803613184}]
I20260807 06:26:51.690747 11986 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:26:51.690943 11986 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:26:51.693578 11986 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e94a63b40cfc4d33ad1365330a7f1fa9: Sending DeleteTablet for 3 replicas of tablet 890e02b1642849678939b558a26ae07e
I20260807 06:26:51.694442 12336 tablet_service.cc:1514] Processing DeleteTablet for tablet 890e02b1642849678939b558a26ae07e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:26:51 UTC) from {username='slave'} at 127.0.0.1:45566
I20260807 06:26:51.694446 12510 tablet_service.cc:1514] Processing DeleteTablet for tablet 890e02b1642849678939b558a26ae07e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:26:51 UTC) from {username='slave'} at 127.0.0.1:46884
I20260807 06:26:51.694525 12158 tablet_service.cc:1514] Processing DeleteTablet for tablet 890e02b1642849678939b558a26ae07e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:26:51 UTC) from {username='slave'} at 127.0.0.1:43416
I20260807 06:26:51.695185 12665 tablet_replica.cc:330] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc: stopping tablet replica
I20260807 06:26:51.695319 12668 tablet_replica.cc:330] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27: stopping tablet replica
I20260807 06:26:51.695434 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 12068
I20260807 06:26:51.695583 12665 raft_consensus.cc:2229] T 890e02b1642849678939b558a26ae07e P 5469444bc2954c3c94429286108561fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:26:51.695612 12668 raft_consensus.cc:2229] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:26:51.695948 12668 raft_consensus.cc:2258] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:26:51.696121 12666 tablet_replica.cc:330] T 890e02b1642849678939b558a26ae07e P e12d0cb79b834e36b71d745ca40bede7: stopping tablet replica
I20260807 06:26:51.696413 12666 raft_consensus.cc:2229] T 890e02b1642849678939b558a26ae07e P e12d0cb79b834e36b71d745ca40bede7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:26:51.696770 12666 raft_consensus.cc:2258] T 890e02b1642849678939b558a26ae07e P e12d0cb79b834e36b71d745ca40bede7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:26:51.702351 12668 ts_tablet_manager.cc:1905] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:26:51.704216 12666 ts_tablet_manager.cc:1905] T 890e02b1642849678939b558a26ae07e P e12d0cb79b834e36b71d745ca40bede7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:26:51.705600 12668 ts_tablet_manager.cc:1918] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:26:51.705796 12668 log.cc:1196] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-1/wal/wals/890e02b1642849678939b558a26ae07e
I20260807 06:26:51.706254 12668 ts_tablet_manager.cc:1939] T 890e02b1642849678939b558a26ae07e P ccc5039faea74d6c89665bdf217e2d27: Deleting consensus metadata
I20260807 06:26:51.707718 11966 catalog_manager.cc:4841] TS ccc5039faea74d6c89665bdf217e2d27 (127.15.254.2:32871): tablet 890e02b1642849678939b558a26ae07e (table test-table [id=2c405db6348244f0ad0de0e13aed1f45]) successfully deleted
I20260807 06:26:51.707862 12666 ts_tablet_manager.cc:1918] T 890e02b1642849678939b558a26ae07e P e12d0cb79b834e36b71d745ca40bede7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:26:51.707983 12666 log.cc:1196] T 890e02b1642849678939b558a26ae07e P e12d0cb79b834e36b71d745ca40bede7: Deleting WAL directory at /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786083738959718-16376-0/minicluster-data/ts-2/wal/wals/890e02b1642849678939b558a26ae07e
I20260807 06:26:51.708355 12666 ts_tablet_manager.cc:1939] T 890e02b1642849678939b558a26ae07e P e12d0cb79b834e36b71d745ca40bede7: Deleting consensus metadata
I20260807 06:26:51.709839 11968 catalog_manager.cc:4841] TS e12d0cb79b834e36b71d745ca40bede7 (127.15.254.3:34421): tablet 890e02b1642849678939b558a26ae07e (table test-table [id=2c405db6348244f0ad0de0e13aed1f45]) successfully deleted
W20260807 06:26:51.711113 11965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.254.1:35049 (error 108)
I20260807 06:26:51.712114 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 12256
I20260807 06:26:51.712347 12624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:51.711966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.1:35049 (local address 127.0.0.1:43420)
0807 06:26:51.712138 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0807 06:26:51.712234 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.1:35049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260807 06:26:51.723310 11965 catalog_manager.cc:4584] TS 5469444bc2954c3c94429286108561fc (127.15.254.1:35049): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 890e02b1642849678939b558a26ae07e: Network error: Client connection negotiation failed: client connection to 127.15.254.1:35049: connect: Connection refused (error 111)
I20260807 06:26:51.731040 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 12428
I20260807 06:26:51.740917 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 11939
2026-08-07T06:26:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1367 ms)
[----------] 6 tests from EncryptionPolicyTest (12090 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:26:51 dist-test-slave-44d2 krb5kdc[12672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:26:51 dist-test-slave-44d2 krb5kdc[12672](info): set up 2 sockets
Aug 07 06:26:51 dist-test-slave-44d2 krb5kdc[12672](info): commencing operation
krb5kdc: starting...
W20260807 06:26:52.973886 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.203s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:26:53 dist-test-slave-44d2 krb5kdc[12672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:26:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:26:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 06:26:53.109947 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46839
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:26:53.225178 12688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:53.225454 12688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:53.225495 12688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:53.229475 12688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:26:53.229542 12688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:53.229560 12688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:53.229578 12688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:26:53.229592 12688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:26:53.234920 12688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46839
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:53.235975 12688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:53.237011 12688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:53.245869 12696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:53.245949 12700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:53.246073 12688 server_base.cc:1034] running on GCE node
W20260807 06:26:53.245869 12698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:53.246690 12688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:53.247807 12688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:53.249009 12688 hybrid_clock.cc:648] HybridClock initialized: now 1786084013248979 us; error 47 us; skew 500 ppm
Aug 07 06:26:53 dist-test-slave-44d2 krb5kdc[12672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:26:53.251928 12688 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 06:26:53.252897 12688 webserver.cc:460] Webserver started at http://127.0.0.1:37443/ using document root <none> and password file <none>
I20260807 06:26:53.253487 12688 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:53.253582 12688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:53.253818 12688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:53.255870 12688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "4525946c4ece4ff3a2d7112beef437fe"
format_stamp: "Formatted at 2026-08-07 06:26:53 on dist-test-slave-44d2"
server_key: "0b66aab8dbf967cdc0563d5807626836"
server_key_iv: "6050755e7f79c52794323f544e257174"
server_key_version: "encryptionkey@0"
I20260807 06:26:53.256429 12688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "4525946c4ece4ff3a2d7112beef437fe"
format_stamp: "Formatted at 2026-08-07 06:26:53 on dist-test-slave-44d2"
server_key: "0b66aab8dbf967cdc0563d5807626836"
server_key_iv: "6050755e7f79c52794323f544e257174"
server_key_version: "encryptionkey@0"
I20260807 06:26:53.260293 12688 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260807 06:26:53.262913 12709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:53.264252 12688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:26:53.264391 12688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "4525946c4ece4ff3a2d7112beef437fe"
format_stamp: "Formatted at 2026-08-07 06:26:53 on dist-test-slave-44d2"
server_key: "0b66aab8dbf967cdc0563d5807626836"
server_key_iv: "6050755e7f79c52794323f544e257174"
server_key_version: "encryptionkey@0"
I20260807 06:26:53.264529 12688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:53.280640 12688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:53.283860 12688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:53.284087 12688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:53.291599 12688 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46839
I20260807 06:26:53.291625 12786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46839 every 8 connection(s)
I20260807 06:26:53.292626 12688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:26:53.295409 12787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:53.297820 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 12688
I20260807 06:26:53.297945 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:26:53.298264 16376 external_mini_cluster.cc:1428] Setting key 214c8092f1d34de7ea7c17722d48421c
I20260807 06:26:53.303574 12787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4525946c4ece4ff3a2d7112beef437fe: Bootstrap starting.
I20260807 06:26:53.306617 12787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4525946c4ece4ff3a2d7112beef437fe: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:53.307441 12787 log.cc:824] T 00000000000000000000000000000000 P 4525946c4ece4ff3a2d7112beef437fe: Log is configured to *not* fsync() on all Append() calls
Aug 07 06:26:53 dist-test-slave-44d2 krb5kdc[12672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 06:26:53.309788 12787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4525946c4ece4ff3a2d7112beef437fe: No bootstrap required, opened a new log
I20260807 06:26:53.314705 12787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4525946c4ece4ff3a2d7112beef437fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4525946c4ece4ff3a2d7112beef437fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46839 } }
I20260807 06:26:53.314990 12787 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4525946c4ece4ff3a2d7112beef437fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:53.315042 12787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4525946c4ece4ff3a2d7112beef437fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4525946c4ece4ff3a2d7112beef437fe, State: Initialized, Role: FOLLOWER
I20260807 06:26:53.315268 12792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:53.299860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34029 (local address 127.0.0.1:46839)
0807 06:26:53.300336 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:53.300359 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:53.300414 (+ 55us) server_negotiation.cc:407] Connection header received
0807 06:26:53.301344 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:53.301371 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:53.301861 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:53.302272 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:53.302952 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:53.304654 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:53.305561 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:53.306276 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:53.306486 (+ 210us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:53.309315 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:53.309322 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:53.309338 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:53.309370 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:53.311554 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:53.312145 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:53.312155 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:53.312163 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:53.312335 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:53.312872 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:53.312875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:53.312877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:53.313393 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:53.313644 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:53.313998 (+ 354us) server_negotiation.cc:299] Negotiation successful
0807 06:26:53.314242 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":124,"threads_started":1}
I20260807 06:26:53.315538 12787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4525946c4ece4ff3a2d7112beef437fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4525946c4ece4ff3a2d7112beef437fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46839 } }
I20260807 06:26:53.315702 12787 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4525946c4ece4ff3a2d7112beef437fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:26:53.315773 12787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4525946c4ece4ff3a2d7112beef437fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:26:53.315865 12787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4525946c4ece4ff3a2d7112beef437fe [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:53.317003 12787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4525946c4ece4ff3a2d7112beef437fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4525946c4ece4ff3a2d7112beef437fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46839 } }
I20260807 06:26:53.317407 12787 leader_election.cc:302] T 00000000000000000000000000000000 P 4525946c4ece4ff3a2d7112beef437fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4525946c4ece4ff3a2d7112beef437fe; no voters:
I20260807 06:26:53.317741 12787 leader_election.cc:288] T 00000000000000000000000000000000 P 4525946c4ece4ff3a2d7112beef437fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:26:53.317860 12794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4525946c4ece4ff3a2d7112beef437fe [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:26:53.318089 12794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4525946c4ece4ff3a2d7112beef437fe [term 1 LEADER]: Becoming Leader. State: Replica: 4525946c4ece4ff3a2d7112beef437fe, State: Running, Role: LEADER
I20260807 06:26:53.318437 12794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4525946c4ece4ff3a2d7112beef437fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4525946c4ece4ff3a2d7112beef437fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46839 } }
I20260807 06:26:53.318609 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:26:53.319099 12787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4525946c4ece4ff3a2d7112beef437fe [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:26:53.320379 12795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4525946c4ece4ff3a2d7112beef437fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4525946c4ece4ff3a2d7112beef437fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4525946c4ece4ff3a2d7112beef437fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46839 } } }
I20260807 06:26:53.320415 12796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4525946c4ece4ff3a2d7112beef437fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4525946c4ece4ff3a2d7112beef437fe. Latest consensus state: current_term: 1 leader_uuid: "4525946c4ece4ff3a2d7112beef437fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4525946c4ece4ff3a2d7112beef437fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46839 } } }
I20260807 06:26:53.320541 12795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4525946c4ece4ff3a2d7112beef437fe [sys.catalog]: This master's current role is: LEADER
I20260807 06:26:53.320547 12796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4525946c4ece4ff3a2d7112beef437fe [sys.catalog]: This master's current role is: LEADER
I20260807 06:26:53.320968 12812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:26:53.324465 12812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:26:53.331028 12812 catalog_manager.cc:1347] Generated new cluster ID: 7f066fe0a16b43a084bccc884b9858f9
I20260807 06:26:53.331094 12812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:26:53.331075 12792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:53.321154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34151 (local address 127.0.0.1:46839)
0807 06:26:53.321254 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:53.321259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:53.321850 (+ 591us) server_negotiation.cc:407] Connection header received
0807 06:26:53.322061 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:53.322065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:53.322117 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:53.322200 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:53.323095 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:53.323843 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:53.325182 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:53.325524 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:53.325599 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:53.327382 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:53.327385 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:53.327388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:53.327417 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:53.329063 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:53.329613 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:53.329617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:53.329619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:53.329707 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:53.330085 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:53.330089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:53.330091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:53.330298 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:53.330457 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:53.330811 (+ 354us) server_negotiation.cc:299] Negotiation successful
0807 06:26:53.330913 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260807 06:26:53.341811 12812 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:26:53.343086 12812 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:26:53.348523 12812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4525946c4ece4ff3a2d7112beef437fe: Generated new TSK 0
I20260807 06:26:53.349359 12812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:26:53.356020 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 12688
W20260807 06:26:53.372182 12670 connection.cc:535] client connection to 127.0.0.1:46839 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46839): Connection reset by peer (error 104)
2026-08-07T06:26:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1616 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:26:53 dist-test-slave-44d2 krb5kdc[12830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:26:53 dist-test-slave-44d2 krb5kdc[12830](info): set up 2 sockets
Aug 07 06:26:53 dist-test-slave-44d2 krb5kdc[12830](info): commencing operation
krb5kdc: starting...
W20260807 06:26:54.610998 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.224s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:26:54 dist-test-slave-44d2 krb5kdc[12830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:26:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:26:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 06:26:54.756073 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40655
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:26:54.874001 12850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:54.874310 12850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:54.874401 12850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:54.878466 12850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:26:54.878556 12850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:54.878603 12850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:54.878659 12850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:26:54.878695 12850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:26:54.884065 12850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40655
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:54.885223 12850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:54.886332 12850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:54.895378 12857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:54.895524 12850 server_base.cc:1034] running on GCE node
W20260807 06:26:54.895378 12856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:54.895421 12860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:54.896317 12850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:54.897496 12850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:54.898698 12850 hybrid_clock.cc:648] HybridClock initialized: now 1786084014898684 us; error 26 us; skew 500 ppm
Aug 07 06:26:54 dist-test-slave-44d2 krb5kdc[12830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:26:54.901599 12850 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 06:26:54.902534 12850 webserver.cc:460] Webserver started at http://127.0.0.1:34015/ using document root <none> and password file <none>
I20260807 06:26:54.903163 12850 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:54.903258 12850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:54.903527 12850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:54.905644 12850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "d7579bcdc8df49d0af1e4d85a410f2b9"
format_stamp: "Formatted at 2026-08-07 06:26:54 on dist-test-slave-44d2"
server_key: "95a120a9aab7cb1c048eab8a4adf76b6"
server_key_iv: "f5b61366013c15cd3240964d5e18758e"
server_key_version: "encryptionkey@0"
I20260807 06:26:54.906232 12850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "d7579bcdc8df49d0af1e4d85a410f2b9"
format_stamp: "Formatted at 2026-08-07 06:26:54 on dist-test-slave-44d2"
server_key: "95a120a9aab7cb1c048eab8a4adf76b6"
server_key_iv: "f5b61366013c15cd3240964d5e18758e"
server_key_version: "encryptionkey@0"
I20260807 06:26:54.910138 12850 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:26:54.912703 12872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:54.914036 12850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 06:26:54.914191 12850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "d7579bcdc8df49d0af1e4d85a410f2b9"
format_stamp: "Formatted at 2026-08-07 06:26:54 on dist-test-slave-44d2"
server_key: "95a120a9aab7cb1c048eab8a4adf76b6"
server_key_iv: "f5b61366013c15cd3240964d5e18758e"
server_key_version: "encryptionkey@0"
I20260807 06:26:54.914309 12850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:54.939682 12850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:54.943159 12850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:54.943401 12850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:54.951388 12850 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40655
I20260807 06:26:54.951471 12940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40655 every 8 connection(s)
I20260807 06:26:54.952507 12850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:26:54.954216 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 12850
I20260807 06:26:54.954463 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:26:54.954819 16376 external_mini_cluster.cc:1428] Setting key bf8b0a83809de1362ea481a060f55c9c
I20260807 06:26:54.956008 12941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:54.963470 12941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7579bcdc8df49d0af1e4d85a410f2b9: Bootstrap starting.
I20260807 06:26:54.966063 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:54.956436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35823 (local address 127.0.0.1:40655)
0807 06:26:54.956978 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:54.957011 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:54.957055 (+ 44us) server_negotiation.cc:407] Connection header received
0807 06:26:54.957926 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:54.957951 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:54.958414 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:54.958829 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:54.959273 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:54.960781 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:54.961826 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:54.962555 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:54.962726 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:54.963876 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:54.963882 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:54.963898 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:26:54.964412 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:54.964523 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:26:54.964570 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:54.964710 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:54.964950 (+ 240us) server_negotiation.cc:299] Negotiation successful
0807 06:26:54.965110 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":131,"threads_started":1}
I20260807 06:26:54.966521 12941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7579bcdc8df49d0af1e4d85a410f2b9: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:54.967321 12941 log.cc:824] T 00000000000000000000000000000000 P d7579bcdc8df49d0af1e4d85a410f2b9: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:54.969449 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:26:54.969810 12941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7579bcdc8df49d0af1e4d85a410f2b9: No bootstrap required, opened a new log
I20260807 06:26:54.974231 12941 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7579bcdc8df49d0af1e4d85a410f2b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7579bcdc8df49d0af1e4d85a410f2b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40655 } }
I20260807 06:26:54.974573 12941 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7579bcdc8df49d0af1e4d85a410f2b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:54.974674 12941 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7579bcdc8df49d0af1e4d85a410f2b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7579bcdc8df49d0af1e4d85a410f2b9, State: Initialized, Role: FOLLOWER
I20260807 06:26:54.975214 12941 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7579bcdc8df49d0af1e4d85a410f2b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7579bcdc8df49d0af1e4d85a410f2b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40655 } }
I20260807 06:26:54.975407 12941 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7579bcdc8df49d0af1e4d85a410f2b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:26:54.975481 12941 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7579bcdc8df49d0af1e4d85a410f2b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:26:54.975606 12941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7579bcdc8df49d0af1e4d85a410f2b9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:54.976461 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:54.971748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50753 (local address 127.0.0.1:40655)
0807 06:26:54.971873 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:54.971878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:54.972136 (+ 258us) server_negotiation.cc:407] Connection header received
0807 06:26:54.972260 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:54.972262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:54.972305 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:54.972380 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:54.972725 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:54.973287 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:54.974401 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:54.974784 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:54.974892 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:54.975880 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:54.975884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:54.975886 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:26:54.975907 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:54.975939 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:26:54.975955 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:54.976036 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:54.976203 (+ 167us) server_negotiation.cc:299] Negotiation successful
0807 06:26:54.976266 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 06:26:54.976994 12941 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7579bcdc8df49d0af1e4d85a410f2b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7579bcdc8df49d0af1e4d85a410f2b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40655 } }
I20260807 06:26:54.977392 12941 leader_election.cc:302] T 00000000000000000000000000000000 P d7579bcdc8df49d0af1e4d85a410f2b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7579bcdc8df49d0af1e4d85a410f2b9; no voters:
I20260807 06:26:54.977775 12941 leader_election.cc:288] T 00000000000000000000000000000000 P d7579bcdc8df49d0af1e4d85a410f2b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:26:54.977885 12956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7579bcdc8df49d0af1e4d85a410f2b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:26:54.978188 12956 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7579bcdc8df49d0af1e4d85a410f2b9 [term 1 LEADER]: Becoming Leader. State: Replica: d7579bcdc8df49d0af1e4d85a410f2b9, State: Running, Role: LEADER
I20260807 06:26:54.978641 12956 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7579bcdc8df49d0af1e4d85a410f2b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7579bcdc8df49d0af1e4d85a410f2b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40655 } }
I20260807 06:26:54.979102 12941 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7579bcdc8df49d0af1e4d85a410f2b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:26:54.980614 12958 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7579bcdc8df49d0af1e4d85a410f2b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7579bcdc8df49d0af1e4d85a410f2b9. Latest consensus state: current_term: 1 leader_uuid: "d7579bcdc8df49d0af1e4d85a410f2b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7579bcdc8df49d0af1e4d85a410f2b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40655 } } }
I20260807 06:26:54.980603 12957 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7579bcdc8df49d0af1e4d85a410f2b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7579bcdc8df49d0af1e4d85a410f2b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7579bcdc8df49d0af1e4d85a410f2b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40655 } } }
I20260807 06:26:54.980772 12957 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7579bcdc8df49d0af1e4d85a410f2b9 [sys.catalog]: This master's current role is: LEADER
I20260807 06:26:54.980772 12958 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7579bcdc8df49d0af1e4d85a410f2b9 [sys.catalog]: This master's current role is: LEADER
I20260807 06:26:54.982089 12970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:26:54.985868 12970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:26:54.991979 12970 catalog_manager.cc:1347] Generated new cluster ID: a9add511d305457db693ae81db3b5a51
I20260807 06:26:54.992067 12970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:26:55.003746 12970 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:26:55.004851 12970 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:26:55.009176 12970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7579bcdc8df49d0af1e4d85a410f2b9: Generated new TSK 0
I20260807 06:26:55.009989 12970 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:26:55.015954 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 12850
W20260807 06:26:55.029106 12827 connection.cc:535] client connection to 127.0.0.1:40655 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40655): Connection reset by peer (error 104)
2026-08-07T06:26:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1656 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:26:55 dist-test-slave-44d2 krb5kdc[12985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:26:55 dist-test-slave-44d2 krb5kdc[12985](info): set up 2 sockets
Aug 07 06:26:55 dist-test-slave-44d2 krb5kdc[12985](info): commencing operation
krb5kdc: starting...
W20260807 06:26:56.242794 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:26:56 dist-test-slave-44d2 krb5kdc[12985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:26:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:26:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 06:26:56.376250 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35931
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:26:56.492722 13005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:56.493027 13005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:56.493100 13005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:56.497150 13005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:26:56.497231 13005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:56.497268 13005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:56.497321 13005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:26:56.497368 13005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:26:56.502627 13005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35931
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:56.503755 13005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:56.504878 13005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:56.513921 13017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:56.513936 13013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:56.513936 13014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:56.513988 13005 server_base.cc:1034] running on GCE node
I20260807 06:26:56.514665 13005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:56.515825 13005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:56.517048 13005 hybrid_clock.cc:648] HybridClock initialized: now 1786084016517035 us; error 30 us; skew 500 ppm
Aug 07 06:26:56 dist-test-slave-44d2 krb5kdc[12985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:26:56.525976 13005 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 06:26:56.526942 13005 webserver.cc:460] Webserver started at http://127.0.0.1:43303/ using document root <none> and password file <none>
I20260807 06:26:56.527518 13005 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:56.527585 13005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:56.527763 13005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:56.529946 13005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "7e2fcf19dd4a46b68e53728883748ce7"
format_stamp: "Formatted at 2026-08-07 06:26:56 on dist-test-slave-44d2"
server_key: "3aca354d4c927532ae17424e3e3076c2"
server_key_iv: "a08d6cba204b159b45c76dca030a2e62"
server_key_version: "encryptionkey@0"
I20260807 06:26:56.530479 13005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "7e2fcf19dd4a46b68e53728883748ce7"
format_stamp: "Formatted at 2026-08-07 06:26:56 on dist-test-slave-44d2"
server_key: "3aca354d4c927532ae17424e3e3076c2"
server_key_iv: "a08d6cba204b159b45c76dca030a2e62"
server_key_version: "encryptionkey@0"
I20260807 06:26:56.534354 13005 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260807 06:26:56.536810 13027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:56.538228 13005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 06:26:56.538350 13005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "7e2fcf19dd4a46b68e53728883748ce7"
format_stamp: "Formatted at 2026-08-07 06:26:56 on dist-test-slave-44d2"
server_key: "3aca354d4c927532ae17424e3e3076c2"
server_key_iv: "a08d6cba204b159b45c76dca030a2e62"
server_key_version: "encryptionkey@0"
I20260807 06:26:56.538442 13005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:56.543150 13005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:56.546561 13005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:56.546713 13005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:56.554442 13005 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35931
I20260807 06:26:56.554459 13096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35931 every 8 connection(s)
I20260807 06:26:56.555392 13005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:26:56.558343 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:56.563393 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 13005
I20260807 06:26:56.563567 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:26:56.563848 16376 external_mini_cluster.cc:1428] Setting key 10e01f6766b85f18843d6864141a5ce8
I20260807 06:26:56.564013 13098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e2fcf19dd4a46b68e53728883748ce7: Bootstrap starting.
I20260807 06:26:56.567034 13098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e2fcf19dd4a46b68e53728883748ce7: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:56.567938 13098 log.cc:824] T 00000000000000000000000000000000 P 7e2fcf19dd4a46b68e53728883748ce7: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:56.570166 13098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e2fcf19dd4a46b68e53728883748ce7: No bootstrap required, opened a new log
I20260807 06:26:56.575124 13098 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e2fcf19dd4a46b68e53728883748ce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e2fcf19dd4a46b68e53728883748ce7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35931 } }
I20260807 06:26:56.575410 13098 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e2fcf19dd4a46b68e53728883748ce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:56.575526 13098 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e2fcf19dd4a46b68e53728883748ce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e2fcf19dd4a46b68e53728883748ce7, State: Initialized, Role: FOLLOWER
I20260807 06:26:56.576116 13098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e2fcf19dd4a46b68e53728883748ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e2fcf19dd4a46b68e53728883748ce7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35931 } }
I20260807 06:26:56.576333 13098 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e2fcf19dd4a46b68e53728883748ce7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:26:56.576406 13098 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e2fcf19dd4a46b68e53728883748ce7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:26:56.576532 13098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e2fcf19dd4a46b68e53728883748ce7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:56.577489 13098 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e2fcf19dd4a46b68e53728883748ce7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e2fcf19dd4a46b68e53728883748ce7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35931 } }
I20260807 06:26:56.577713 13103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:56.565869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60795 (local address 127.0.0.1:35931)
0807 06:26:56.572237 (+ 6368us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:56.572257 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:56.572327 (+ 70us) server_negotiation.cc:407] Connection header received
0807 06:26:56.574729 (+ 2402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:56.574749 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:56.575055 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:56.575356 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:56.576306 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:56.576314 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:56.576330 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:26:56.576343 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:56.576440 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:26:56.576472 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:56.576632 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:56.576738 (+ 106us) server_negotiation.cc:299] Negotiation successful
0807 06:26:56.576902 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6158,"thread_start_us":164,"threads_started":1}
I20260807 06:26:56.577852 13098 leader_election.cc:302] T 00000000000000000000000000000000 P 7e2fcf19dd4a46b68e53728883748ce7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e2fcf19dd4a46b68e53728883748ce7; no voters:
I20260807 06:26:56.578207 13098 leader_election.cc:288] T 00000000000000000000000000000000 P 7e2fcf19dd4a46b68e53728883748ce7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:26:56.578289 13107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e2fcf19dd4a46b68e53728883748ce7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:26:56.578496 13107 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e2fcf19dd4a46b68e53728883748ce7 [term 1 LEADER]: Becoming Leader. State: Replica: 7e2fcf19dd4a46b68e53728883748ce7, State: Running, Role: LEADER
I20260807 06:26:56.578856 13107 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e2fcf19dd4a46b68e53728883748ce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e2fcf19dd4a46b68e53728883748ce7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35931 } }
I20260807 06:26:56.579516 13098 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e2fcf19dd4a46b68e53728883748ce7 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:26:56.580073 13107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e2fcf19dd4a46b68e53728883748ce7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e2fcf19dd4a46b68e53728883748ce7. Latest consensus state: current_term: 1 leader_uuid: "7e2fcf19dd4a46b68e53728883748ce7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e2fcf19dd4a46b68e53728883748ce7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35931 } } }
I20260807 06:26:56.580188 13107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e2fcf19dd4a46b68e53728883748ce7 [sys.catalog]: This master's current role is: LEADER
I20260807 06:26:56.580511 13108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e2fcf19dd4a46b68e53728883748ce7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e2fcf19dd4a46b68e53728883748ce7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e2fcf19dd4a46b68e53728883748ce7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35931 } } }
I20260807 06:26:56.580669 13108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e2fcf19dd4a46b68e53728883748ce7 [sys.catalog]: This master's current role is: LEADER
I20260807 06:26:56.581014 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260807 06:26:56.584911 13124 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e2fcf19dd4a46b68e53728883748ce7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:26:56.585052 13124 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:26:56.585180 13113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:26:56.585686 13103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:56.583337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59657 (local address 127.0.0.1:35931)
0807 06:26:56.583447 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:56.583451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:56.583669 (+ 218us) server_negotiation.cc:407] Connection header received
0807 06:26:56.583821 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:56.583824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:56.583862 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:56.583945 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:56.585109 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:56.585114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:56.585117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:26:56.585132 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:56.585172 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:26:56.585192 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:56.585306 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:56.585493 (+ 187us) server_negotiation.cc:299] Negotiation successful
0807 06:26:56.585544 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260807 06:26:56.586349 13113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:26:56.592463 13113 catalog_manager.cc:1347] Generated new cluster ID: 12cad36603fc4106b0c36351e3de795a
I20260807 06:26:56.592602 13113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:26:56.617528 13113 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:26:56.617887 13113 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:26:56.625651 13113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e2fcf19dd4a46b68e53728883748ce7: Generated new TSK 0
I20260807 06:26:56.626335 13113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:26:56.633222 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 13005
2026-08-07T06:26:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1619 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:26:56 dist-test-slave-44d2 krb5kdc[13137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:26:56 dist-test-slave-44d2 krb5kdc[13137](info): set up 2 sockets
Aug 07 06:26:56 dist-test-slave-44d2 krb5kdc[13137](info): commencing operation
krb5kdc: starting...
W20260807 06:26:57.860438 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.197s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:26:57 dist-test-slave-44d2 krb5kdc[13137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:26:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:26:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 06:26:58.000103 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36297
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:26:58.115221 13161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:58.115545 13161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:58.115610 13161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:58.119668 13161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:26:58.119764 13161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:58.119802 13161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:58.119863 13161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:26:58.119902 13161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:26:58.125201 13161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:58.126374 13161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:58.127472 13161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:58.136548 13170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:58.136557 13167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:58.136557 13168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:58.136710 13161 server_base.cc:1034] running on GCE node
I20260807 06:26:58.137329 13161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:58.138461 13161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:58.139657 13161 hybrid_clock.cc:648] HybridClock initialized: now 1786084018139649 us; error 28 us; skew 500 ppm
Aug 07 06:26:58 dist-test-slave-44d2 krb5kdc[13137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:26:58.142449 13161 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 06:26:58.143350 13161 webserver.cc:460] Webserver started at http://127.0.0.1:41447/ using document root <none> and password file <none>
I20260807 06:26:58.143878 13161 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:58.143939 13161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:58.144114 13161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:58.146207 13161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "7786ec30ed0042efb1cc66b56916111e"
format_stamp: "Formatted at 2026-08-07 06:26:58 on dist-test-slave-44d2"
server_key: "39418087a2768ed217f3b72406a90e74"
server_key_iv: "fd8b8bb37af420bc5e0708b83ee5e029"
server_key_version: "encryptionkey@0"
I20260807 06:26:58.146716 13161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "7786ec30ed0042efb1cc66b56916111e"
format_stamp: "Formatted at 2026-08-07 06:26:58 on dist-test-slave-44d2"
server_key: "39418087a2768ed217f3b72406a90e74"
server_key_iv: "fd8b8bb37af420bc5e0708b83ee5e029"
server_key_version: "encryptionkey@0"
I20260807 06:26:58.150584 13161 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260807 06:26:58.153039 13180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:58.154276 13161 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 06:26:58.154410 13161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "7786ec30ed0042efb1cc66b56916111e"
format_stamp: "Formatted at 2026-08-07 06:26:58 on dist-test-slave-44d2"
server_key: "39418087a2768ed217f3b72406a90e74"
server_key_iv: "fd8b8bb37af420bc5e0708b83ee5e029"
server_key_version: "encryptionkey@0"
I20260807 06:26:58.154513 13161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:58.165161 13161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:58.168272 13161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:58.168543 13161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:58.176319 13161 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36297
I20260807 06:26:58.176332 13243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36297 every 8 connection(s)
I20260807 06:26:58.177320 13161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:26:58.177876 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 13161
I20260807 06:26:58.178081 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:26:58.178403 16376 external_mini_cluster.cc:1428] Setting key 136baaad885ca4f83dd99d0e2c83245e
I20260807 06:26:58.180686 13244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:58.187474 13244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7786ec30ed0042efb1cc66b56916111e: Bootstrap starting.
Aug 07 06:26:58 dist-test-slave-44d2 krb5kdc[13137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 06:26:58.190388 13244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7786ec30ed0042efb1cc66b56916111e: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:58.191368 13244 log.cc:824] T 00000000000000000000000000000000 P 7786ec30ed0042efb1cc66b56916111e: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:58.193974 13244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7786ec30ed0042efb1cc66b56916111e: No bootstrap required, opened a new log
I20260807 06:26:58.194792 13247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:58.179920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58453 (local address 127.0.0.1:36297)
0807 06:26:58.180372 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:58.180391 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:58.180435 (+ 44us) server_negotiation.cc:407] Connection header received
0807 06:26:58.181298 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:58.181321 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:58.181744 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:58.182115 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:58.182798 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:58.184428 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:58.185343 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:58.186074 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:58.186239 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:58.189202 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:58.189209 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:58.189224 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:58.189257 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:58.191272 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:58.191828 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:58.191834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:58.191842 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:58.191963 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:58.192396 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:58.192400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:58.192402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:58.192978 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:58.193192 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:58.193460 (+ 268us) server_negotiation.cc:299] Negotiation successful
0807 06:26:58.193763 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":117,"threads_started":1}
I20260807 06:26:58.198077 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:26:58.198401 13244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7786ec30ed0042efb1cc66b56916111e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7786ec30ed0042efb1cc66b56916111e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36297 } }
I20260807 06:26:58.198618 13244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7786ec30ed0042efb1cc66b56916111e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:58.198690 13244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7786ec30ed0042efb1cc66b56916111e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7786ec30ed0042efb1cc66b56916111e, State: Initialized, Role: FOLLOWER
I20260807 06:26:58.199146 13244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7786ec30ed0042efb1cc66b56916111e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7786ec30ed0042efb1cc66b56916111e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36297 } }
I20260807 06:26:58.199295 13244 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7786ec30ed0042efb1cc66b56916111e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:26:58.199445 13244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7786ec30ed0042efb1cc66b56916111e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:26:58.199581 13244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7786ec30ed0042efb1cc66b56916111e [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:58.200973 13244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7786ec30ed0042efb1cc66b56916111e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7786ec30ed0042efb1cc66b56916111e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36297 } }
I20260807 06:26:58.201361 13244 leader_election.cc:302] T 00000000000000000000000000000000 P 7786ec30ed0042efb1cc66b56916111e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7786ec30ed0042efb1cc66b56916111e; no voters:
I20260807 06:26:58.201676 13244 leader_election.cc:288] T 00000000000000000000000000000000 P 7786ec30ed0042efb1cc66b56916111e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:26:58.201821 13255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7786ec30ed0042efb1cc66b56916111e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:26:58.202129 13255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7786ec30ed0042efb1cc66b56916111e [term 1 LEADER]: Becoming Leader. State: Replica: 7786ec30ed0042efb1cc66b56916111e, State: Running, Role: LEADER
I20260807 06:26:58.202478 13255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7786ec30ed0042efb1cc66b56916111e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7786ec30ed0042efb1cc66b56916111e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36297 } }
I20260807 06:26:58.203032 13244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7786ec30ed0042efb1cc66b56916111e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:26:58.203733 13255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7786ec30ed0042efb1cc66b56916111e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7786ec30ed0042efb1cc66b56916111e. Latest consensus state: current_term: 1 leader_uuid: "7786ec30ed0042efb1cc66b56916111e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7786ec30ed0042efb1cc66b56916111e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36297 } } }
I20260807 06:26:58.203943 13255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7786ec30ed0042efb1cc66b56916111e [sys.catalog]: This master's current role is: LEADER
I20260807 06:26:58.204212 13260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7786ec30ed0042efb1cc66b56916111e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7786ec30ed0042efb1cc66b56916111e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7786ec30ed0042efb1cc66b56916111e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36297 } } }
I20260807 06:26:58.204371 13272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:26:58.204394 13260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7786ec30ed0042efb1cc66b56916111e [sys.catalog]: This master's current role is: LEADER
I20260807 06:26:58.208670 13272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:26:58.210598 13247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:58.200545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60021 (local address 127.0.0.1:36297)
0807 06:26:58.200687 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:58.200691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:58.200733 (+ 42us) server_negotiation.cc:407] Connection header received
0807 06:26:58.200906 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:58.200913 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:58.200968 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:58.201077 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:58.201988 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:58.202722 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:58.203910 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:58.204279 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:58.204388 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:58.206211 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:58.206218 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:58.206221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:26:58.206250 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:58.208074 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:58.208605 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:58.208612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:58.208614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:58.208718 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:26:58.209126 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:26:58.209130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:26:58.209131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:26:58.209369 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:58.209502 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:58.210298 (+ 796us) server_negotiation.cc:299] Negotiation successful
0807 06:26:58.210427 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 06:26:58.217175 13272 catalog_manager.cc:1347] Generated new cluster ID: fc9f05d503f747d18d325a0451535d39
I20260807 06:26:58.217258 13272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:26:58.227080 13272 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:26:58.228049 13272 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:26:58.233769 13272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7786ec30ed0042efb1cc66b56916111e: Generated new TSK 0
I20260807 06:26:58.234462 13272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:26:58.238278 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 13161
2026-08-07T06:26:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1606 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:26:58 dist-test-slave-44d2 krb5kdc[13287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:26:58 dist-test-slave-44d2 krb5kdc[13287](info): set up 2 sockets
Aug 07 06:26:58 dist-test-slave-44d2 krb5kdc[13287](info): commencing operation
krb5kdc: starting...
W20260807 06:26:59.473078 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.204s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:26:59 dist-test-slave-44d2 krb5kdc[13287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:26:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:26:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 06:26:59.614233 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39987
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:26:59.728837 13303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:26:59.729211 13303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:26:59.729305 13303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:26:59.733600 13303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:26:59.733697 13303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:26:59.733737 13303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:26:59.733798 13303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:26:59.733839 13303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:26:59.739289 13303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:26:59.740414 13303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:26:59.741505 13303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:26:59.750423 13312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:26:59.750437 13316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:59.750638 13303 server_base.cc:1034] running on GCE node
W20260807 06:26:59.750443 13313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:26:59.751401 13303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:26:59.752671 13303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:26:59.753875 13303 hybrid_clock.cc:648] HybridClock initialized: now 1786084019753854 us; error 33 us; skew 500 ppm
Aug 07 06:26:59 dist-test-slave-44d2 krb5kdc[13287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:26:59.756899 13303 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 06:26:59.757897 13303 webserver.cc:460] Webserver started at http://127.0.0.1:34505/ using document root <none> and password file <none>
I20260807 06:26:59.758500 13303 fs_manager.cc:359] Metadata directory not provided
I20260807 06:26:59.758598 13303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:26:59.758849 13303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:26:59.761070 13303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "4b6e4cd3d9d34624ade212a10f2af212"
format_stamp: "Formatted at 2026-08-07 06:26:59 on dist-test-slave-44d2"
server_key: "b63e95404d824f04313d051eff373781"
server_key_iv: "7a7146a8500675cd999e87ac6b6457d6"
server_key_version: "encryptionkey@0"
I20260807 06:26:59.761668 13303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "4b6e4cd3d9d34624ade212a10f2af212"
format_stamp: "Formatted at 2026-08-07 06:26:59 on dist-test-slave-44d2"
server_key: "b63e95404d824f04313d051eff373781"
server_key_iv: "7a7146a8500675cd999e87ac6b6457d6"
server_key_version: "encryptionkey@0"
I20260807 06:26:59.765518 13303 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:26:59.767977 13325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:26:59.769433 13303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:26:59.769598 13303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "4b6e4cd3d9d34624ade212a10f2af212"
format_stamp: "Formatted at 2026-08-07 06:26:59 on dist-test-slave-44d2"
server_key: "b63e95404d824f04313d051eff373781"
server_key_iv: "7a7146a8500675cd999e87ac6b6457d6"
server_key_version: "encryptionkey@0"
I20260807 06:26:59.769722 13303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:26:59.785564 13303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:26:59.788802 13303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:26:59.789047 13303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:26:59.797254 13303 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39987
I20260807 06:26:59.797290 13384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39987 every 8 connection(s)
I20260807 06:26:59.798295 13303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:26:59.801419 13386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:26:59.801898 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 13303
I20260807 06:26:59.802057 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:26:59.802379 16376 external_mini_cluster.cc:1428] Setting key 9c14bf6a67a8652e1b172f34d51d1dab
I20260807 06:26:59.809469 13386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b6e4cd3d9d34624ade212a10f2af212: Bootstrap starting.
I20260807 06:26:59.812682 13386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b6e4cd3d9d34624ade212a10f2af212: Neither blocks nor log segments found. Creating new log.
I20260807 06:26:59.813714 13389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:59.803905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57543 (local address 127.0.0.1:39987)
0807 06:26:59.804362 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:59.804384 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:59.804433 (+ 49us) server_negotiation.cc:407] Connection header received
0807 06:26:59.805293 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:59.805319 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:59.805800 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:59.806202 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:59.806643 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:59.808351 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:59.809474 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:59.810213 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:59.810390 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:59.811394 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:59.811401 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:59.811418 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:26:59.811954 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:59.812031 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:26:59.812066 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:59.812178 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:59.812285 (+ 107us) server_negotiation.cc:299] Negotiation successful
0807 06:26:59.812601 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":113,"threads_started":1}
I20260807 06:26:59.813679 13386 log.cc:824] T 00000000000000000000000000000000 P 4b6e4cd3d9d34624ade212a10f2af212: Log is configured to *not* fsync() on all Append() calls
I20260807 06:26:59.816282 13386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b6e4cd3d9d34624ade212a10f2af212: No bootstrap required, opened a new log
I20260807 06:26:59.816885 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:26:59.821676 13386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b6e4cd3d9d34624ade212a10f2af212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6e4cd3d9d34624ade212a10f2af212" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39987 } }
I20260807 06:26:59.822074 13386 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b6e4cd3d9d34624ade212a10f2af212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:26:59.822145 13386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b6e4cd3d9d34624ade212a10f2af212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b6e4cd3d9d34624ade212a10f2af212, State: Initialized, Role: FOLLOWER
I20260807 06:26:59.822690 13386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b6e4cd3d9d34624ade212a10f2af212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6e4cd3d9d34624ade212a10f2af212" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39987 } }
I20260807 06:26:59.822901 13386 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b6e4cd3d9d34624ade212a10f2af212 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:26:59.822988 13386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b6e4cd3d9d34624ade212a10f2af212 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:26:59.823132 13386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b6e4cd3d9d34624ade212a10f2af212 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:26:59.823742 13389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:26:59.819302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55737 (local address 127.0.0.1:39987)
0807 06:26:59.819447 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0807 06:26:59.819452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:26:59.819598 (+ 146us) server_negotiation.cc:407] Connection header received
0807 06:26:59.819686 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:26:59.819691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:26:59.819741 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:26:59.819857 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:26:59.820081 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:59.820998 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:59.821822 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:26:59.822218 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:26:59.822350 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:26:59.823293 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:26:59.823297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:26:59.823299 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:26:59.823313 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:26:59.823348 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:26:59.823360 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:26:59.823442 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 06:26:59.823545 (+ 103us) server_negotiation.cc:299] Negotiation successful
0807 06:26:59.823604 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 06:26:59.824553 13386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b6e4cd3d9d34624ade212a10f2af212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6e4cd3d9d34624ade212a10f2af212" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39987 } }
I20260807 06:26:59.825018 13386 leader_election.cc:302] T 00000000000000000000000000000000 P 4b6e4cd3d9d34624ade212a10f2af212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b6e4cd3d9d34624ade212a10f2af212; no voters:
I20260807 06:26:59.825384 13386 leader_election.cc:288] T 00000000000000000000000000000000 P 4b6e4cd3d9d34624ade212a10f2af212 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:26:59.825507 13399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b6e4cd3d9d34624ade212a10f2af212 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:26:59.825753 13399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b6e4cd3d9d34624ade212a10f2af212 [term 1 LEADER]: Becoming Leader. State: Replica: 4b6e4cd3d9d34624ade212a10f2af212, State: Running, Role: LEADER
I20260807 06:26:59.826141 13399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b6e4cd3d9d34624ade212a10f2af212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6e4cd3d9d34624ade212a10f2af212" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39987 } }
I20260807 06:26:59.826920 13386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b6e4cd3d9d34624ade212a10f2af212 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:26:59.827831 13401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b6e4cd3d9d34624ade212a10f2af212 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b6e4cd3d9d34624ade212a10f2af212. Latest consensus state: current_term: 1 leader_uuid: "4b6e4cd3d9d34624ade212a10f2af212" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6e4cd3d9d34624ade212a10f2af212" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39987 } } }
I20260807 06:26:59.827962 13401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b6e4cd3d9d34624ade212a10f2af212 [sys.catalog]: This master's current role is: LEADER
I20260807 06:26:59.828302 13400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b6e4cd3d9d34624ade212a10f2af212 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b6e4cd3d9d34624ade212a10f2af212" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6e4cd3d9d34624ade212a10f2af212" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39987 } } }
I20260807 06:26:59.828532 13400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b6e4cd3d9d34624ade212a10f2af212 [sys.catalog]: This master's current role is: LEADER
I20260807 06:26:59.829455 13410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:26:59.832870 13410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:26:59.838898 13410 catalog_manager.cc:1347] Generated new cluster ID: ff237b5410e44e828ffa0162b2576821
I20260807 06:26:59.838980 13410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:26:59.853753 13410 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:26:59.855010 13410 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:26:59.864224 13410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b6e4cd3d9d34624ade212a10f2af212: Generated new TSK 0
I20260807 06:26:59.865063 13410 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:26:59.870178 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 13303
2026-08-07T06:26:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1633 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:26:59 dist-test-slave-44d2 krb5kdc[13427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:26:59 dist-test-slave-44d2 krb5kdc[13427](info): set up 2 sockets
Aug 07 06:26:59 dist-test-slave-44d2 krb5kdc[13427](info): commencing operation
krb5kdc: starting...
W20260807 06:27:01.122458 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.219s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:27:01 dist-test-slave-44d2 krb5kdc[13427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:27:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:27:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 06:27:01.264590 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39637
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:27:01.393854 13448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:27:01.394418 13448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:27:01.394559 13448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:27:01.401124 13448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:27:01.401209 13448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:27:01.401243 13448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:27:01.401297 13448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:27:01.401328 13448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:27:01.412410 13448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:27:01.414096 13448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:27:01.415637 13448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:27:01.426440 13458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:01.426440 13456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:01.426469 13455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:27:01.426813 13448 server_base.cc:1034] running on GCE node
I20260807 06:27:01.427461 13448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:27:01.428808 13448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:27:01.430033 13448 hybrid_clock.cc:648] HybridClock initialized: now 1786084021429998 us; error 48 us; skew 500 ppm
Aug 07 06:27:01 dist-test-slave-44d2 krb5kdc[13427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:27:01.433111 13448 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 06:27:01.434058 13448 webserver.cc:460] Webserver started at http://127.0.0.1:37787/ using document root <none> and password file <none>
I20260807 06:27:01.434685 13448 fs_manager.cc:359] Metadata directory not provided
I20260807 06:27:01.434751 13448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:27:01.435024 13448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:27:01.437500 13448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "5ffaed38ce32489f8e0f7d60f73d4431"
format_stamp: "Formatted at 2026-08-07 06:27:01 on dist-test-slave-44d2"
server_key: "8b7aa75727ac3589fe3f59438e91eb37"
server_key_iv: "95558d61a0580a957b5b81a9db8af581"
server_key_version: "encryptionkey@0"
I20260807 06:27:01.438155 13448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "5ffaed38ce32489f8e0f7d60f73d4431"
format_stamp: "Formatted at 2026-08-07 06:27:01 on dist-test-slave-44d2"
server_key: "8b7aa75727ac3589fe3f59438e91eb37"
server_key_iv: "95558d61a0580a957b5b81a9db8af581"
server_key_version: "encryptionkey@0"
I20260807 06:27:01.442188 13448 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260807 06:27:01.444922 13468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:27:01.446319 13448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:27:01.446476 13448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "5ffaed38ce32489f8e0f7d60f73d4431"
format_stamp: "Formatted at 2026-08-07 06:27:01 on dist-test-slave-44d2"
server_key: "8b7aa75727ac3589fe3f59438e91eb37"
server_key_iv: "95558d61a0580a957b5b81a9db8af581"
server_key_version: "encryptionkey@0"
I20260807 06:27:01.446590 13448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:27:01.451534 13448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:27:01.454917 13448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:27:01.455128 13448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:27:01.463290 13448 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39637
I20260807 06:27:01.463306 13538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39637 every 8 connection(s)
I20260807 06:27:01.464751 13448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:27:01.467710 13540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:27:01.473198 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 13448
I20260807 06:27:01.473399 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:27:01.473677 13540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ffaed38ce32489f8e0f7d60f73d4431: Bootstrap starting.
I20260807 06:27:01.473735 16376 external_mini_cluster.cc:1428] Setting key a1508d7d0d861fa3d4157369a4bbc11d
I20260807 06:27:01.476338 13540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ffaed38ce32489f8e0f7d60f73d4431: Neither blocks nor log segments found. Creating new log.
I20260807 06:27:01.477260 13540 log.cc:824] T 00000000000000000000000000000000 P 5ffaed38ce32489f8e0f7d60f73d4431: Log is configured to *not* fsync() on all Append() calls
I20260807 06:27:01.479440 13540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ffaed38ce32489f8e0f7d60f73d4431: No bootstrap required, opened a new log
I20260807 06:27:01.479655 13543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:01.475247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34463 (local address 127.0.0.1:39637)
0807 06:27:01.475743 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:01.475761 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:01.475800 (+ 39us) server_negotiation.cc:407] Connection header received
0807 06:27:01.476659 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:01.476682 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:01.477007 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:01.477391 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:01.478280 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:01.478285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:01.478298 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:27:01.478310 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:01.478369 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:27:01.478390 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:01.478529 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:01.478649 (+ 120us) server_negotiation.cc:299] Negotiation successful
0807 06:27:01.478830 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":144,"threads_started":1}
I20260807 06:27:01.482960 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:27:01.483736 13540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ffaed38ce32489f8e0f7d60f73d4431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ffaed38ce32489f8e0f7d60f73d4431" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39637 } }
I20260807 06:27:01.483994 13540 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ffaed38ce32489f8e0f7d60f73d4431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:27:01.484088 13540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ffaed38ce32489f8e0f7d60f73d4431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ffaed38ce32489f8e0f7d60f73d4431, State: Initialized, Role: FOLLOWER
I20260807 06:27:01.484632 13540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ffaed38ce32489f8e0f7d60f73d4431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ffaed38ce32489f8e0f7d60f73d4431" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39637 } }
I20260807 06:27:01.484813 13540 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ffaed38ce32489f8e0f7d60f73d4431 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:27:01.484882 13540 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ffaed38ce32489f8e0f7d60f73d4431 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:27:01.484982 13540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ffaed38ce32489f8e0f7d60f73d4431 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:27:01.486389 13540 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ffaed38ce32489f8e0f7d60f73d4431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ffaed38ce32489f8e0f7d60f73d4431" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39637 } }
I20260807 06:27:01.486801 13540 leader_election.cc:302] T 00000000000000000000000000000000 P 5ffaed38ce32489f8e0f7d60f73d4431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ffaed38ce32489f8e0f7d60f73d4431; no voters:
I20260807 06:27:01.487187 13540 leader_election.cc:288] T 00000000000000000000000000000000 P 5ffaed38ce32489f8e0f7d60f73d4431 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:27:01.487375 13543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:01.485413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45725 (local address 127.0.0.1:39637)
0807 06:27:01.485563 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:01.485568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:01.485782 (+ 214us) server_negotiation.cc:407] Connection header received
0807 06:27:01.485892 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:01.485895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:01.485956 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:01.486030 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:01.486841 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:01.486845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:01.486847 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:27:01.486858 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:01.486885 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:27:01.486895 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:01.487062 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:01.487178 (+ 116us) server_negotiation.cc:299] Negotiation successful
0807 06:27:01.487228 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 06:27:01.487480 13550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ffaed38ce32489f8e0f7d60f73d4431 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:27:01.487797 13550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ffaed38ce32489f8e0f7d60f73d4431 [term 1 LEADER]: Becoming Leader. State: Replica: 5ffaed38ce32489f8e0f7d60f73d4431, State: Running, Role: LEADER
I20260807 06:27:01.488171 13550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ffaed38ce32489f8e0f7d60f73d4431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ffaed38ce32489f8e0f7d60f73d4431" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39637 } }
I20260807 06:27:01.488797 13540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ffaed38ce32489f8e0f7d60f73d4431 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:27:01.490193 13553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ffaed38ce32489f8e0f7d60f73d4431 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ffaed38ce32489f8e0f7d60f73d4431. Latest consensus state: current_term: 1 leader_uuid: "5ffaed38ce32489f8e0f7d60f73d4431" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ffaed38ce32489f8e0f7d60f73d4431" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39637 } } }
I20260807 06:27:01.490324 13553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ffaed38ce32489f8e0f7d60f73d4431 [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:01.490665 13551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ffaed38ce32489f8e0f7d60f73d4431 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ffaed38ce32489f8e0f7d60f73d4431" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ffaed38ce32489f8e0f7d60f73d4431" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39637 } } }
I20260807 06:27:01.490762 13551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ffaed38ce32489f8e0f7d60f73d4431 [sys.catalog]: This master's current role is: LEADER
W20260807 06:27:01.494105 13567 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ffaed38ce32489f8e0f7d60f73d4431: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:27:01.494176 13567 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:27:01.494239 13562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:27:01.494987 13562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:27:01.501199 13562 catalog_manager.cc:1347] Generated new cluster ID: e640e0207bec4e62a8cce81e343391c1
I20260807 06:27:01.501289 13562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:27:01.530339 13562 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:27:01.530771 13562 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:27:01.540613 13562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ffaed38ce32489f8e0f7d60f73d4431: Generated new TSK 0
I20260807 06:27:01.541541 13562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:27:01.551038 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 13448
2026-08-07T06:27:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1675 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:27:01 dist-test-slave-44d2 krb5kdc[13574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:27:01 dist-test-slave-44d2 krb5kdc[13574](info): set up 2 sockets
Aug 07 06:27:01 dist-test-slave-44d2 krb5kdc[13574](info): commencing operation
krb5kdc: starting...
W20260807 06:27:02.803010 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.223s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:27:02 dist-test-slave-44d2 krb5kdc[13574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:27:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:27:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 06:27:02.956298 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41165
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:27:03.087220 13593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:27:03.087546 13593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:27:03.087626 13593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:27:03.091698 13593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:27:03.091792 13593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:27:03.091876 13593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:27:03.091940 13593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:27:03.091981 13593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:27:03.097465 13593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:27:03.098652 13593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:27:03.099759 13593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:27:03.108831 13605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:03.108831 13603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:27:03.109077 13593 server_base.cc:1034] running on GCE node
W20260807 06:27:03.109326 13601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:27:03.110000 13593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:27:03.111255 13593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:27:03.112468 13593 hybrid_clock.cc:648] HybridClock initialized: now 1786084023112444 us; error 35 us; skew 500 ppm
Aug 07 06:27:03 dist-test-slave-44d2 krb5kdc[13574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:27:03.115803 13593 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 06:27:03.116910 13593 webserver.cc:460] Webserver started at http://127.0.0.1:45973/ using document root <none> and password file <none>
I20260807 06:27:03.117554 13593 fs_manager.cc:359] Metadata directory not provided
I20260807 06:27:03.117658 13593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:27:03.117923 13593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:27:03.120232 13593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "befea892c842417e801caf8873082675"
format_stamp: "Formatted at 2026-08-07 06:27:03 on dist-test-slave-44d2"
server_key: "6e96ea214c5fb7637673194dfb45aad4"
server_key_iv: "ba0b38986d058765efccb24ac267af70"
server_key_version: "encryptionkey@0"
I20260807 06:27:03.120924 13593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "befea892c842417e801caf8873082675"
format_stamp: "Formatted at 2026-08-07 06:27:03 on dist-test-slave-44d2"
server_key: "6e96ea214c5fb7637673194dfb45aad4"
server_key_iv: "ba0b38986d058765efccb24ac267af70"
server_key_version: "encryptionkey@0"
I20260807 06:27:03.125241 13593 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 06:27:03.128173 13616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:27:03.129804 13593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:27:03.129988 13593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "befea892c842417e801caf8873082675"
format_stamp: "Formatted at 2026-08-07 06:27:03 on dist-test-slave-44d2"
server_key: "6e96ea214c5fb7637673194dfb45aad4"
server_key_iv: "ba0b38986d058765efccb24ac267af70"
server_key_version: "encryptionkey@0"
I20260807 06:27:03.130141 13593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:27:03.146639 13593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:27:03.150391 13593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:27:03.150635 13593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:27:03.158668 13593 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41165
I20260807 06:27:03.158696 13692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41165 every 8 connection(s)
I20260807 06:27:03.159749 13593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:27:03.162827 13693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:27:03.165676 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 13593
I20260807 06:27:03.165858 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:27:03.166194 16376 external_mini_cluster.cc:1428] Setting key 44bcc00b66759d495c593367d16f80fe
I20260807 06:27:03.169974 13693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P befea892c842417e801caf8873082675: Bootstrap starting.
I20260807 06:27:03.172950 13693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P befea892c842417e801caf8873082675: Neither blocks nor log segments found. Creating new log.
I20260807 06:27:03.173995 13693 log.cc:824] T 00000000000000000000000000000000 P befea892c842417e801caf8873082675: Log is configured to *not* fsync() on all Append() calls
I20260807 06:27:03.176847 13693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P befea892c842417e801caf8873082675: No bootstrap required, opened a new log
Aug 07 06:27:03 dist-test-slave-44d2 krb5kdc[13574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 06:27:03.180692 13693 raft_consensus.cc:348] T 00000000000000000000000000000000 P befea892c842417e801caf8873082675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "befea892c842417e801caf8873082675" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41165 } }
I20260807 06:27:03.180977 13693 raft_consensus.cc:374] T 00000000000000000000000000000000 P befea892c842417e801caf8873082675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:27:03.181047 13693 raft_consensus.cc:729] T 00000000000000000000000000000000 P befea892c842417e801caf8873082675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: befea892c842417e801caf8873082675, State: Initialized, Role: FOLLOWER
I20260807 06:27:03.181558 13693 consensus_queue.cc:260] T 00000000000000000000000000000000 P befea892c842417e801caf8873082675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "befea892c842417e801caf8873082675" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41165 } }
I20260807 06:27:03.181695 13693 raft_consensus.cc:388] T 00000000000000000000000000000000 P befea892c842417e801caf8873082675 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:27:03.181793 13693 raft_consensus.cc:482] T 00000000000000000000000000000000 P befea892c842417e801caf8873082675 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:27:03.181947 13693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P befea892c842417e801caf8873082675 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:27:03.182962 13693 raft_consensus.cc:504] T 00000000000000000000000000000000 P befea892c842417e801caf8873082675 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "befea892c842417e801caf8873082675" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41165 } }
I20260807 06:27:03.183352 13693 leader_election.cc:302] T 00000000000000000000000000000000 P befea892c842417e801caf8873082675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: befea892c842417e801caf8873082675; no voters:
I20260807 06:27:03.183772 13693 leader_election.cc:288] T 00000000000000000000000000000000 P befea892c842417e801caf8873082675 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:27:03.183832 13705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P befea892c842417e801caf8873082675 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:27:03.184132 13705 raft_consensus.cc:686] T 00000000000000000000000000000000 P befea892c842417e801caf8873082675 [term 1 LEADER]: Becoming Leader. State: Replica: befea892c842417e801caf8873082675, State: Running, Role: LEADER
I20260807 06:27:03.184588 13705 consensus_queue.cc:237] T 00000000000000000000000000000000 P befea892c842417e801caf8873082675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "befea892c842417e801caf8873082675" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41165 } }
I20260807 06:27:03.185421 13693 sys_catalog.cc:564] T 00000000000000000000000000000000 P befea892c842417e801caf8873082675 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:27:03.185473 13700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:03.167976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:48683 (local address 127.0.0.1:41165)
0807 06:27:03.168598 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:03.168619 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:03.168662 (+ 43us) server_negotiation.cc:407] Connection header received
0807 06:27:03.169412 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:03.169431 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:03.169834 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:03.170292 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:03.171281 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:03.172447 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:03.173935 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:03.174633 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:03.174821 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:03.178414 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:03.178421 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:03.178437 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:03.178522 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:03.181680 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:03.182442 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:03.182449 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:03.182455 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:03.182592 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:03.183002 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:03.183006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:03.183007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:03.183586 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:03.183839 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:03.184237 (+ 398us) server_negotiation.cc:299] Negotiation successful
0807 06:27:03.184564 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":157,"threads_started":1}
I20260807 06:27:03.185942 13705 sys_catalog.cc:455] T 00000000000000000000000000000000 P befea892c842417e801caf8873082675 [sys.catalog]: SysCatalogTable state changed. Reason: New leader befea892c842417e801caf8873082675. Latest consensus state: current_term: 1 leader_uuid: "befea892c842417e801caf8873082675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "befea892c842417e801caf8873082675" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41165 } } }
I20260807 06:27:03.186079 13705 sys_catalog.cc:458] T 00000000000000000000000000000000 P befea892c842417e801caf8873082675 [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:03.186311 13705 sys_catalog.cc:455] T 00000000000000000000000000000000 P befea892c842417e801caf8873082675 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "befea892c842417e801caf8873082675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "befea892c842417e801caf8873082675" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41165 } } }
I20260807 06:27:03.186512 13705 sys_catalog.cc:458] T 00000000000000000000000000000000 P befea892c842417e801caf8873082675 [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:03.186939 13709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:27:03.189307 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:27:03.192194 13709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:27:03.199786 13709 catalog_manager.cc:1347] Generated new cluster ID: a9add59c8fda421aaa1ce0f7d3d818f3
I20260807 06:27:03.199868 13709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:27:03.203508 13700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:03.191907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:59645 (local address 127.0.0.1:41165)
0807 06:27:03.192029 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:03.192033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:03.192868 (+ 835us) server_negotiation.cc:407] Connection header received
0807 06:27:03.193092 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:03.193096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:03.193152 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:03.193263 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:03.194241 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:03.194944 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:03.196359 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:03.196758 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:03.196877 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:03.198608 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:03.198613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:03.198616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:03.198645 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:03.201293 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:03.202044 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:03.202048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:03.202050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:03.202143 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:03.202532 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:03.202535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:03.202543 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:03.202745 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:03.202886 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:03.203238 (+ 352us) server_negotiation.cc:299] Negotiation successful
0807 06:27:03.203340 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260807 06:27:03.211778 13709 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:27:03.212884 13709 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:27:03.219548 13709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P befea892c842417e801caf8873082675: Generated new TSK 0
I20260807 06:27:03.220275 13709 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:27:03.227087 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 13593
W20260807 06:27:03.244633 13572 connection.cc:535] client connection to 127.0.0.1:41165 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41165): Connection reset by peer (error 104)
2026-08-07T06:27:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1680 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:27:03 dist-test-slave-44d2 krb5kdc[13736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:27:03 dist-test-slave-44d2 krb5kdc[13736](info): set up 2 sockets
Aug 07 06:27:03 dist-test-slave-44d2 krb5kdc[13736](info): commencing operation
krb5kdc: starting...
W20260807 06:27:04.544229 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.284s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:27:04 dist-test-slave-44d2 krb5kdc[13736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:27:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:27:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 06:27:04.691610 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37455
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:27:04.808537 13754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:27:04.808830 13754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:27:04.808873 13754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:27:04.812896 13754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:27:04.812971 13754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:27:04.812991 13754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:27:04.813007 13754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:27:04.813022 13754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:27:04.818470 13754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:27:04.819588 13754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:27:04.820644 13754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:27:04.829730 13760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:04.829737 13763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:04.829774 13761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:27:04.830122 13754 server_base.cc:1034] running on GCE node
I20260807 06:27:04.830783 13754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:27:04.831997 13754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:27:04.833202 13754 hybrid_clock.cc:648] HybridClock initialized: now 1786084024833174 us; error 53 us; skew 500 ppm
Aug 07 06:27:04 dist-test-slave-44d2 krb5kdc[13736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:27:04.836134 13754 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 06:27:04.837165 13754 webserver.cc:460] Webserver started at http://127.0.0.1:39975/ using document root <none> and password file <none>
I20260807 06:27:04.837780 13754 fs_manager.cc:359] Metadata directory not provided
I20260807 06:27:04.837879 13754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:27:04.838146 13754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:27:04.840385 13754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "6179e241095a4879867275e42a7c543d"
format_stamp: "Formatted at 2026-08-07 06:27:04 on dist-test-slave-44d2"
server_key: "66a5236cb6b5198967367c6813c20536"
server_key_iv: "34cff3940e730d5ef1a306b29ab664e7"
server_key_version: "encryptionkey@0"
I20260807 06:27:04.841059 13754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "6179e241095a4879867275e42a7c543d"
format_stamp: "Formatted at 2026-08-07 06:27:04 on dist-test-slave-44d2"
server_key: "66a5236cb6b5198967367c6813c20536"
server_key_iv: "34cff3940e730d5ef1a306b29ab664e7"
server_key_version: "encryptionkey@0"
I20260807 06:27:04.844969 13754 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:27:04.847608 13772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:27:04.848977 13754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 06:27:04.849138 13754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "6179e241095a4879867275e42a7c543d"
format_stamp: "Formatted at 2026-08-07 06:27:04 on dist-test-slave-44d2"
server_key: "66a5236cb6b5198967367c6813c20536"
server_key_iv: "34cff3940e730d5ef1a306b29ab664e7"
server_key_version: "encryptionkey@0"
I20260807 06:27:04.849254 13754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:27:04.876087 13754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:27:04.879769 13754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:27:04.880031 13754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:27:04.889385 13754 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37455
I20260807 06:27:04.889459 13844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37455 every 8 connection(s)
I20260807 06:27:04.890561 13754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:27:04.893571 13845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:27:04.899339 13845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6179e241095a4879867275e42a7c543d: Bootstrap starting.
I20260807 06:27:04.899976 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 13754
I20260807 06:27:04.900174 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:27:04.900552 16376 external_mini_cluster.cc:1428] Setting key 4c8f09469c9f33a34d1c564239e82f1c
I20260807 06:27:04.902508 13845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6179e241095a4879867275e42a7c543d: Neither blocks nor log segments found. Creating new log.
I20260807 06:27:04.903579 13845 log.cc:824] T 00000000000000000000000000000000 P 6179e241095a4879867275e42a7c543d: Log is configured to *not* fsync() on all Append() calls
I20260807 06:27:04.906189 13845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6179e241095a4879867275e42a7c543d: No bootstrap required, opened a new log
I20260807 06:27:04.910522 13845 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6179e241095a4879867275e42a7c543d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6179e241095a4879867275e42a7c543d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37455 } }
I20260807 06:27:04.910815 13845 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6179e241095a4879867275e42a7c543d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:27:04.910872 13845 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6179e241095a4879867275e42a7c543d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6179e241095a4879867275e42a7c543d, State: Initialized, Role: FOLLOWER
I20260807 06:27:04.911401 13845 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6179e241095a4879867275e42a7c543d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6179e241095a4879867275e42a7c543d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37455 } }
I20260807 06:27:04.911574 13845 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6179e241095a4879867275e42a7c543d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:27:04.911659 13845 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6179e241095a4879867275e42a7c543d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:27:04.911772 13845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6179e241095a4879867275e42a7c543d [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:27:04.912436 13849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:04.901878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:55645 (local address 127.0.0.1:37455)
0807 06:27:04.902697 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:04.902721 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:04.902772 (+ 51us) server_negotiation.cc:407] Connection header received
0807 06:27:04.903794 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:04.903824 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:04.904334 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:04.904776 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:04.905695 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:04.907192 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:04.908218 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:04.909038 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:04.909243 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:04.910172 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:04.910179 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:04.910195 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:27:04.910701 (+ 506us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:04.910777 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:27:04.910811 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:04.910954 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:04.911206 (+ 252us) server_negotiation.cc:299] Negotiation successful
0807 06:27:04.911342 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":200,"threads_started":1}
I20260807 06:27:04.912992 13845 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6179e241095a4879867275e42a7c543d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6179e241095a4879867275e42a7c543d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37455 } }
I20260807 06:27:04.913350 13845 leader_election.cc:302] T 00000000000000000000000000000000 P 6179e241095a4879867275e42a7c543d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6179e241095a4879867275e42a7c543d; no voters:
I20260807 06:27:04.913763 13845 leader_election.cc:288] T 00000000000000000000000000000000 P 6179e241095a4879867275e42a7c543d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:27:04.913823 13851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6179e241095a4879867275e42a7c543d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:27:04.914186 13851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6179e241095a4879867275e42a7c543d [term 1 LEADER]: Becoming Leader. State: Replica: 6179e241095a4879867275e42a7c543d, State: Running, Role: LEADER
I20260807 06:27:04.914973 13851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6179e241095a4879867275e42a7c543d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6179e241095a4879867275e42a7c543d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37455 } }
I20260807 06:27:04.915177 13845 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6179e241095a4879867275e42a7c543d [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:27:04.915828 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:27:04.916301 13851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6179e241095a4879867275e42a7c543d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6179e241095a4879867275e42a7c543d. Latest consensus state: current_term: 1 leader_uuid: "6179e241095a4879867275e42a7c543d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6179e241095a4879867275e42a7c543d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37455 } } }
I20260807 06:27:04.916531 13851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6179e241095a4879867275e42a7c543d [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:04.916821 13853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6179e241095a4879867275e42a7c543d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6179e241095a4879867275e42a7c543d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6179e241095a4879867275e42a7c543d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37455 } } }
I20260807 06:27:04.916932 13853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6179e241095a4879867275e42a7c543d [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:04.917047 13863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:27:04.920641 13863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:27:04.925899 13849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:04.918387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:53107 (local address 127.0.0.1:37455)
0807 06:27:04.918520 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:04.918524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:04.920815 (+ 2291us) server_negotiation.cc:407] Connection header received
0807 06:27:04.921257 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:04.921263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:04.921323 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:04.921375 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:04.921876 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:04.922649 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:04.923900 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:04.924225 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:04.924339 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:04.925300 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:04.925306 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:04.925310 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:27:04.925331 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:04.925365 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:27:04.925383 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:04.925489 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:04.925699 (+ 210us) server_negotiation.cc:299] Negotiation successful
0807 06:27:04.925739 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260807 06:27:04.927393 13863 catalog_manager.cc:1347] Generated new cluster ID: 0a3bfa9de6e540c69a388cd6d4a1b7a6
I20260807 06:27:04.927477 13863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:27:04.941390 13863 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:27:04.942616 13863 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:27:04.955905 13863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6179e241095a4879867275e42a7c543d: Generated new TSK 0
I20260807 06:27:04.956950 13863 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:27:04.961680 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 13754
W20260807 06:27:04.975193 13734 connection.cc:535] client connection to 127.0.0.1:37455 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37455): Connection reset by peer (error 104)
2026-08-07T06:27:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1730 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:27:05 dist-test-slave-44d2 krb5kdc[13882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:27:05 dist-test-slave-44d2 krb5kdc[13882](info): set up 2 sockets
Aug 07 06:27:05 dist-test-slave-44d2 krb5kdc[13882](info): commencing operation
krb5kdc: starting...
W20260807 06:27:06.181278 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.191s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:27:06 dist-test-slave-44d2 krb5kdc[13882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:27:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:27:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 06:27:06.323925 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41097
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:27:06.443382 13900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:27:06.443773 13900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:27:06.443863 13900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:27:06.448107 13900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:27:06.448223 13900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:27:06.448262 13900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:27:06.448304 13900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:27:06.448343 13900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:27:06.454174 13900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:27:06.455389 13900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:27:06.456473 13900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:27:06.465636 13907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:06.465644 13909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:06.465644 13906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:27:06.466034 13900 server_base.cc:1034] running on GCE node
I20260807 06:27:06.466758 13900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:27:06.468245 13900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:27:06.469481 13900 hybrid_clock.cc:648] HybridClock initialized: now 1786084026469443 us; error 53 us; skew 500 ppm
Aug 07 06:27:06 dist-test-slave-44d2 krb5kdc[13882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:27:06.472973 13900 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 06:27:06.474027 13900 webserver.cc:460] Webserver started at http://127.0.0.1:42609/ using document root <none> and password file <none>
I20260807 06:27:06.474679 13900 fs_manager.cc:359] Metadata directory not provided
I20260807 06:27:06.474778 13900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:27:06.475034 13900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:27:06.477306 13900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "1df65028eeb84a6f93af4dbb1415c7ac"
format_stamp: "Formatted at 2026-08-07 06:27:06 on dist-test-slave-44d2"
server_key: "348506cac1d6164454a2f2e2f920e09d"
server_key_iv: "19841643133075f6e1833b07b25742c8"
server_key_version: "encryptionkey@0"
I20260807 06:27:06.477900 13900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "1df65028eeb84a6f93af4dbb1415c7ac"
format_stamp: "Formatted at 2026-08-07 06:27:06 on dist-test-slave-44d2"
server_key: "348506cac1d6164454a2f2e2f920e09d"
server_key_iv: "19841643133075f6e1833b07b25742c8"
server_key_version: "encryptionkey@0"
I20260807 06:27:06.481885 13900 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:27:06.484403 13920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:27:06.485749 13900 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 06:27:06.485917 13900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "1df65028eeb84a6f93af4dbb1415c7ac"
format_stamp: "Formatted at 2026-08-07 06:27:06 on dist-test-slave-44d2"
server_key: "348506cac1d6164454a2f2e2f920e09d"
server_key_iv: "19841643133075f6e1833b07b25742c8"
server_key_version: "encryptionkey@0"
I20260807 06:27:06.486045 13900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:27:06.490844 13900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:27:06.494508 13900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:27:06.494733 13900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:27:06.502748 13900 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41097
I20260807 06:27:06.502770 13992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41097 every 8 connection(s)
I20260807 06:27:06.503866 13900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:27:06.506832 13994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:27:06.512547 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 13900
I20260807 06:27:06.512748 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:27:06.513054 16376 external_mini_cluster.cc:1428] Setting key 1eaf2ce0ebfc3c6e7e88d8c8d30acab7
I20260807 06:27:06.513105 13994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1df65028eeb84a6f93af4dbb1415c7ac: Bootstrap starting.
I20260807 06:27:06.516245 13994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1df65028eeb84a6f93af4dbb1415c7ac: Neither blocks nor log segments found. Creating new log.
I20260807 06:27:06.517292 13994 log.cc:824] T 00000000000000000000000000000000 P 1df65028eeb84a6f93af4dbb1415c7ac: Log is configured to *not* fsync() on all Append() calls
I20260807 06:27:06.519464 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:06.514556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:53133 (local address 127.0.0.1:41097)
0807 06:27:06.515067 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:06.515087 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:06.515134 (+ 47us) server_negotiation.cc:407] Connection header received
0807 06:27:06.516060 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:06.516089 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:06.516542 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:06.516944 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:06.517697 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:06.517704 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:06.517722 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:27:06.517744 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:06.517811 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:27:06.517843 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:06.517937 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:06.518183 (+ 246us) server_negotiation.cc:299] Negotiation successful
0807 06:27:06.518453 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":149,"threads_started":1}
I20260807 06:27:06.519960 13994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1df65028eeb84a6f93af4dbb1415c7ac: No bootstrap required, opened a new log
I20260807 06:27:06.522615 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:27:06.524355 13994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1df65028eeb84a6f93af4dbb1415c7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df65028eeb84a6f93af4dbb1415c7ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41097 } }
I20260807 06:27:06.524724 13994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1df65028eeb84a6f93af4dbb1415c7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:27:06.524850 13994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1df65028eeb84a6f93af4dbb1415c7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df65028eeb84a6f93af4dbb1415c7ac, State: Initialized, Role: FOLLOWER
I20260807 06:27:06.525460 13994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1df65028eeb84a6f93af4dbb1415c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df65028eeb84a6f93af4dbb1415c7ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41097 } }
I20260807 06:27:06.525650 13994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1df65028eeb84a6f93af4dbb1415c7ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:27:06.525743 13994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1df65028eeb84a6f93af4dbb1415c7ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:27:06.525882 13994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1df65028eeb84a6f93af4dbb1415c7ac [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:27:06.526661 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:06.524862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:55409 (local address 127.0.0.1:41097)
0807 06:27:06.524993 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:06.524998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:06.525184 (+ 186us) server_negotiation.cc:407] Connection header received
0807 06:27:06.525348 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:06.525351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:06.525394 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:06.525480 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:06.526255 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:06.526258 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:06.526261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:27:06.526273 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:06.526301 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:27:06.526320 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:06.526412 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:06.526484 (+ 72us) server_negotiation.cc:299] Negotiation successful
0807 06:27:06.526537 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 06:27:06.527081 13994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1df65028eeb84a6f93af4dbb1415c7ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df65028eeb84a6f93af4dbb1415c7ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41097 } }
I20260807 06:27:06.527474 13994 leader_election.cc:302] T 00000000000000000000000000000000 P 1df65028eeb84a6f93af4dbb1415c7ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1df65028eeb84a6f93af4dbb1415c7ac; no voters:
I20260807 06:27:06.527828 13994 leader_election.cc:288] T 00000000000000000000000000000000 P 1df65028eeb84a6f93af4dbb1415c7ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:27:06.527938 14009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1df65028eeb84a6f93af4dbb1415c7ac [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:27:06.528159 14009 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1df65028eeb84a6f93af4dbb1415c7ac [term 1 LEADER]: Becoming Leader. State: Replica: 1df65028eeb84a6f93af4dbb1415c7ac, State: Running, Role: LEADER
I20260807 06:27:06.528469 14009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1df65028eeb84a6f93af4dbb1415c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df65028eeb84a6f93af4dbb1415c7ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41097 } }
I20260807 06:27:06.529354 13994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1df65028eeb84a6f93af4dbb1415c7ac [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:27:06.530478 14010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1df65028eeb84a6f93af4dbb1415c7ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1df65028eeb84a6f93af4dbb1415c7ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df65028eeb84a6f93af4dbb1415c7ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41097 } } }
I20260807 06:27:06.530514 14011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1df65028eeb84a6f93af4dbb1415c7ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1df65028eeb84a6f93af4dbb1415c7ac. Latest consensus state: current_term: 1 leader_uuid: "1df65028eeb84a6f93af4dbb1415c7ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df65028eeb84a6f93af4dbb1415c7ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41097 } } }
I20260807 06:27:06.530618 14010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1df65028eeb84a6f93af4dbb1415c7ac [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:06.530628 14011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1df65028eeb84a6f93af4dbb1415c7ac [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:06.531044 14018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:27:06.534578 14018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:27:06.540903 14018 catalog_manager.cc:1347] Generated new cluster ID: 4e755fec1d3b4e948dfb74896b9f78f4
I20260807 06:27:06.540992 14018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:27:06.565340 14018 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:27:06.565759 14018 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:27:06.582386 14018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1df65028eeb84a6f93af4dbb1415c7ac: Generated new TSK 0
I20260807 06:27:06.583256 14018 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:27:06.598132 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 13900
2026-08-07T06:27:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1635 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:27:06 dist-test-slave-44d2 krb5kdc[14034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:27:06 dist-test-slave-44d2 krb5kdc[14034](info): set up 2 sockets
Aug 07 06:27:06 dist-test-slave-44d2 krb5kdc[14034](info): commencing operation
krb5kdc: starting...
W20260807 06:27:07.833335 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.207s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:27:07 dist-test-slave-44d2 krb5kdc[14034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:27:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:27:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 06:27:07.977413 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43397
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:27:08.097236 14050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:27:08.097548 14050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:27:08.097622 14050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:27:08.101677 14050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:27:08.101760 14050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:27:08.101799 14050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:27:08.101857 14050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:27:08.101895 14050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:27:08.107326 14050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43397
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:27:08.108417 14050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:27:08.109470 14050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:27:08.118436 14063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:27:08.118430 14050 server_base.cc:1034] running on GCE node
W20260807 06:27:08.118489 14061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:08.118409 14060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:27:08.119179 14050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:27:08.120303 14050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:27:08.121512 14050 hybrid_clock.cc:648] HybridClock initialized: now 1786084028121484 us; error 45 us; skew 500 ppm
Aug 07 06:27:08 dist-test-slave-44d2 krb5kdc[14034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:27:08.124361 14050 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 06:27:08.125329 14050 webserver.cc:460] Webserver started at http://127.0.0.1:37157/ using document root <none> and password file <none>
I20260807 06:27:08.125911 14050 fs_manager.cc:359] Metadata directory not provided
I20260807 06:27:08.126004 14050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:27:08.126250 14050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:27:08.128319 14050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "df389a058719417b9ab8b827cd4972cc"
format_stamp: "Formatted at 2026-08-07 06:27:08 on dist-test-slave-44d2"
server_key: "471eb2a0332dc77990ec837eba862c8f"
server_key_iv: "7550ca6e02e1cf8e4004e65cf5bd248d"
server_key_version: "encryptionkey@0"
I20260807 06:27:08.128999 14050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "df389a058719417b9ab8b827cd4972cc"
format_stamp: "Formatted at 2026-08-07 06:27:08 on dist-test-slave-44d2"
server_key: "471eb2a0332dc77990ec837eba862c8f"
server_key_iv: "7550ca6e02e1cf8e4004e65cf5bd248d"
server_key_version: "encryptionkey@0"
I20260807 06:27:08.133023 14050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 06:27:08.135771 14071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:27:08.137250 14050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:27:08.137418 14050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "df389a058719417b9ab8b827cd4972cc"
format_stamp: "Formatted at 2026-08-07 06:27:08 on dist-test-slave-44d2"
server_key: "471eb2a0332dc77990ec837eba862c8f"
server_key_iv: "7550ca6e02e1cf8e4004e65cf5bd248d"
server_key_version: "encryptionkey@0"
I20260807 06:27:08.137539 14050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:27:08.158046 14050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:27:08.161489 14050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:27:08.161712 14050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:27:08.169629 14050 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43397
I20260807 06:27:08.169682 14142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43397 every 8 connection(s)
I20260807 06:27:08.170640 14050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:27:08.173556 14143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:27:08.175657 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 14050
I20260807 06:27:08.175802 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:27:08.176132 16376 external_mini_cluster.cc:1428] Setting key 6d34988a1907ed53bac6a95490ac06a5
I20260807 06:27:08.180305 14143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df389a058719417b9ab8b827cd4972cc: Bootstrap starting.
I20260807 06:27:08.182600 14143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df389a058719417b9ab8b827cd4972cc: Neither blocks nor log segments found. Creating new log.
I20260807 06:27:08.183435 14143 log.cc:824] T 00000000000000000000000000000000 P df389a058719417b9ab8b827cd4972cc: Log is configured to *not* fsync() on all Append() calls
Aug 07 06:27:08 dist-test-slave-44d2 krb5kdc[14034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 06:27:08.185683 14143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df389a058719417b9ab8b827cd4972cc: No bootstrap required, opened a new log
I20260807 06:27:08.189009 14143 raft_consensus.cc:348] T 00000000000000000000000000000000 P df389a058719417b9ab8b827cd4972cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df389a058719417b9ab8b827cd4972cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } }
I20260807 06:27:08.189275 14143 raft_consensus.cc:374] T 00000000000000000000000000000000 P df389a058719417b9ab8b827cd4972cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:27:08.189366 14143 raft_consensus.cc:729] T 00000000000000000000000000000000 P df389a058719417b9ab8b827cd4972cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df389a058719417b9ab8b827cd4972cc, State: Initialized, Role: FOLLOWER
I20260807 06:27:08.189846 14143 consensus_queue.cc:260] T 00000000000000000000000000000000 P df389a058719417b9ab8b827cd4972cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df389a058719417b9ab8b827cd4972cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } }
I20260807 06:27:08.190023 14143 raft_consensus.cc:388] T 00000000000000000000000000000000 P df389a058719417b9ab8b827cd4972cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:27:08.190100 14143 raft_consensus.cc:482] T 00000000000000000000000000000000 P df389a058719417b9ab8b827cd4972cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:27:08.190248 14143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df389a058719417b9ab8b827cd4972cc [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:27:08.191377 14143 raft_consensus.cc:504] T 00000000000000000000000000000000 P df389a058719417b9ab8b827cd4972cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df389a058719417b9ab8b827cd4972cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } }
I20260807 06:27:08.191679 14147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:08.177575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:50265 (local address 127.0.0.1:43397)
0807 06:27:08.178016 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:08.178031 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:08.178090 (+ 59us) server_negotiation.cc:407] Connection header received
0807 06:27:08.178727 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:08.178745 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:08.179087 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:08.179441 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:08.180294 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:08.181381 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:08.182259 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:08.182795 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:08.182963 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:08.185873 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:08.185889 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:08.185904 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:08.185935 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:08.188263 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:08.188878 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:08.188883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:08.188888 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:08.189005 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:08.189413 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:08.189416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:08.189418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:08.189937 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:08.190201 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:08.190530 (+ 329us) server_negotiation.cc:299] Negotiation successful
0807 06:27:08.190822 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":123,"threads_started":1}
I20260807 06:27:08.191768 14143 leader_election.cc:302] T 00000000000000000000000000000000 P df389a058719417b9ab8b827cd4972cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df389a058719417b9ab8b827cd4972cc; no voters:
I20260807 06:27:08.192193 14143 leader_election.cc:288] T 00000000000000000000000000000000 P df389a058719417b9ab8b827cd4972cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:27:08.192301 14149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df389a058719417b9ab8b827cd4972cc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:27:08.192512 14149 raft_consensus.cc:686] T 00000000000000000000000000000000 P df389a058719417b9ab8b827cd4972cc [term 1 LEADER]: Becoming Leader. State: Replica: df389a058719417b9ab8b827cd4972cc, State: Running, Role: LEADER
I20260807 06:27:08.192893 14149 consensus_queue.cc:237] T 00000000000000000000000000000000 P df389a058719417b9ab8b827cd4972cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df389a058719417b9ab8b827cd4972cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } }
I20260807 06:27:08.193601 14143 sys_catalog.cc:564] T 00000000000000000000000000000000 P df389a058719417b9ab8b827cd4972cc [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:27:08.195062 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:27:08.195056 14152 sys_catalog.cc:455] T 00000000000000000000000000000000 P df389a058719417b9ab8b827cd4972cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df389a058719417b9ab8b827cd4972cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df389a058719417b9ab8b827cd4972cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } } }
I20260807 06:27:08.195195 14152 sys_catalog.cc:458] T 00000000000000000000000000000000 P df389a058719417b9ab8b827cd4972cc [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:08.195041 14153 sys_catalog.cc:455] T 00000000000000000000000000000000 P df389a058719417b9ab8b827cd4972cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader df389a058719417b9ab8b827cd4972cc. Latest consensus state: current_term: 1 leader_uuid: "df389a058719417b9ab8b827cd4972cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df389a058719417b9ab8b827cd4972cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } } }
I20260807 06:27:08.195246 14153 sys_catalog.cc:458] T 00000000000000000000000000000000 P df389a058719417b9ab8b827cd4972cc [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:08.195699 14163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:27:08.199262 14163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:27:08.205698 14163 catalog_manager.cc:1347] Generated new cluster ID: 0cbfc04eb1d447b6b9777848c514ff40
I20260807 06:27:08.205837 14163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:27:08.207361 14147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:08.197571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:53519 (local address 127.0.0.1:43397)
0807 06:27:08.197704 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:08.197710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:08.198250 (+ 540us) server_negotiation.cc:407] Connection header received
0807 06:27:08.198365 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:08.198369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:08.198429 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:08.198520 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:08.199371 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:08.200161 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:08.201348 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:08.201595 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:08.201712 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:08.203300 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:08.203304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:08.203308 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:08.203331 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:08.205127 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:08.205881 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:08.205885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:08.205888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:08.205986 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:08.206385 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:08.206389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:08.206391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:08.206629 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:08.206801 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:08.207083 (+ 282us) server_negotiation.cc:299] Negotiation successful
0807 06:27:08.207200 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 06:27:08.217772 14163 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:27:08.218770 14163 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:27:08.225605 14163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df389a058719417b9ab8b827cd4972cc: Generated new TSK 0
I20260807 06:27:08.226454 14163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:27:08.236829 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 14050
W20260807 06:27:08.248104 14032 connection.cc:535] client connection to 127.0.0.1:43397 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43397): Connection reset by peer (error 104)
2026-08-07T06:27:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1637 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:27:08 dist-test-slave-44d2 krb5kdc[14184](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:27:08 dist-test-slave-44d2 krb5kdc[14184](info): set up 2 sockets
Aug 07 06:27:08 dist-test-slave-44d2 krb5kdc[14184](info): commencing operation
krb5kdc: starting...
W20260807 06:27:09.494148 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.231s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:27:09 dist-test-slave-44d2 krb5kdc[14184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:27:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:27:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 06:27:09.636088 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40809
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:27:09.752049 14199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:27:09.752321 14199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:27:09.752362 14199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:27:09.756351 14199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:27:09.756421 14199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:27:09.756441 14199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:27:09.756458 14199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:27:09.756474 14199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:27:09.762039 14199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40809
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:27:09.763129 14199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:27:09.764129 14199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:27:09.773254 14209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:27:09.773486 14199 server_base.cc:1034] running on GCE node
W20260807 06:27:09.773284 14204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:09.773260 14207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:27:09.774142 14199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:27:09.775277 14199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:27:09.776531 14199 hybrid_clock.cc:648] HybridClock initialized: now 1786084029776447 us; error 55 us; skew 500 ppm
Aug 07 06:27:09 dist-test-slave-44d2 krb5kdc[14184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:27:09.779513 14199 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 06:27:09.780467 14199 webserver.cc:460] Webserver started at http://127.0.0.1:41085/ using document root <none> and password file <none>
I20260807 06:27:09.781090 14199 fs_manager.cc:359] Metadata directory not provided
I20260807 06:27:09.781157 14199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:27:09.781332 14199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:27:09.783434 14199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "1a667659df8d4a449b7f8f37e07accd7"
format_stamp: "Formatted at 2026-08-07 06:27:09 on dist-test-slave-44d2"
server_key: "effa35b0f1dfbc855c8d389e77fc6a76"
server_key_iv: "cdbff4fedfda7e4f3765e7092a4e65ea"
server_key_version: "encryptionkey@0"
I20260807 06:27:09.783967 14199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "1a667659df8d4a449b7f8f37e07accd7"
format_stamp: "Formatted at 2026-08-07 06:27:09 on dist-test-slave-44d2"
server_key: "effa35b0f1dfbc855c8d389e77fc6a76"
server_key_iv: "cdbff4fedfda7e4f3765e7092a4e65ea"
server_key_version: "encryptionkey@0"
I20260807 06:27:09.787851 14199 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260807 06:27:09.790472 14218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:27:09.791849 14199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 06:27:09.791975 14199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "1a667659df8d4a449b7f8f37e07accd7"
format_stamp: "Formatted at 2026-08-07 06:27:09 on dist-test-slave-44d2"
server_key: "effa35b0f1dfbc855c8d389e77fc6a76"
server_key_iv: "cdbff4fedfda7e4f3765e7092a4e65ea"
server_key_version: "encryptionkey@0"
I20260807 06:27:09.792071 14199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:27:09.821313 14199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:27:09.825027 14199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:27:09.825205 14199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:27:09.833001 14199 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40809
I20260807 06:27:09.833027 14296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40809 every 8 connection(s)
I20260807 06:27:09.834096 14199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:27:09.837015 14297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:27:09.842732 14297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a667659df8d4a449b7f8f37e07accd7: Bootstrap starting.
I20260807 06:27:09.843969 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 14199
I20260807 06:27:09.844130 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:27:09.844449 16376 external_mini_cluster.cc:1428] Setting key c5d01f9adbf596af76a712b45dd6405c
I20260807 06:27:09.845541 14297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a667659df8d4a449b7f8f37e07accd7: Neither blocks nor log segments found. Creating new log.
I20260807 06:27:09.846442 14297 log.cc:824] T 00000000000000000000000000000000 P 1a667659df8d4a449b7f8f37e07accd7: Log is configured to *not* fsync() on all Append() calls
I20260807 06:27:09.848394 14297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a667659df8d4a449b7f8f37e07accd7: No bootstrap required, opened a new log
I20260807 06:27:09.851923 14297 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a667659df8d4a449b7f8f37e07accd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a667659df8d4a449b7f8f37e07accd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40809 } }
I20260807 06:27:09.852223 14297 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a667659df8d4a449b7f8f37e07accd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:27:09.852324 14297 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a667659df8d4a449b7f8f37e07accd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a667659df8d4a449b7f8f37e07accd7, State: Initialized, Role: FOLLOWER
I20260807 06:27:09.852877 14297 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a667659df8d4a449b7f8f37e07accd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a667659df8d4a449b7f8f37e07accd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40809 } }
I20260807 06:27:09.853036 14297 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a667659df8d4a449b7f8f37e07accd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:27:09.853106 14297 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a667659df8d4a449b7f8f37e07accd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:27:09.853220 14297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a667659df8d4a449b7f8f37e07accd7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:27:09.854233 14297 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a667659df8d4a449b7f8f37e07accd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a667659df8d4a449b7f8f37e07accd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40809 } }
I20260807 06:27:09.854377 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:09.845701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:55517 (local address 127.0.0.1:40809)
0807 06:27:09.846226 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:09.846242 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:09.846281 (+ 39us) server_negotiation.cc:407] Connection header received
0807 06:27:09.847181 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:09.847202 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:09.847567 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:09.847964 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:09.848419 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:09.849564 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:09.850593 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:09.851137 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:09.851326 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:09.852443 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:09.852451 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:09.852474 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:27:09.852918 (+ 444us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:09.852987 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:27:09.853012 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:09.853200 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:09.853486 (+ 286us) server_negotiation.cc:299] Negotiation successful
0807 06:27:09.853625 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":146,"threads_started":1}
I20260807 06:27:09.854605 14297 leader_election.cc:302] T 00000000000000000000000000000000 P 1a667659df8d4a449b7f8f37e07accd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a667659df8d4a449b7f8f37e07accd7; no voters:
I20260807 06:27:09.854969 14297 leader_election.cc:288] T 00000000000000000000000000000000 P 1a667659df8d4a449b7f8f37e07accd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:27:09.855034 14307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a667659df8d4a449b7f8f37e07accd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:27:09.855244 14307 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a667659df8d4a449b7f8f37e07accd7 [term 1 LEADER]: Becoming Leader. State: Replica: 1a667659df8d4a449b7f8f37e07accd7, State: Running, Role: LEADER
I20260807 06:27:09.855602 14307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a667659df8d4a449b7f8f37e07accd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a667659df8d4a449b7f8f37e07accd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40809 } }
I20260807 06:27:09.856309 14297 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a667659df8d4a449b7f8f37e07accd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:27:09.857410 14309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a667659df8d4a449b7f8f37e07accd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a667659df8d4a449b7f8f37e07accd7. Latest consensus state: current_term: 1 leader_uuid: "1a667659df8d4a449b7f8f37e07accd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a667659df8d4a449b7f8f37e07accd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40809 } } }
I20260807 06:27:09.857656 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:27:09.857839 14309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a667659df8d4a449b7f8f37e07accd7 [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:09.857729 14308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a667659df8d4a449b7f8f37e07accd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a667659df8d4a449b7f8f37e07accd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a667659df8d4a449b7f8f37e07accd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40809 } } }
I20260807 06:27:09.858070 14308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a667659df8d4a449b7f8f37e07accd7 [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:09.858322 14316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:27:09.862262 14316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:27:09.865123 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:09.860109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:50049 (local address 127.0.0.1:40809)
0807 06:27:09.860233 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:09.860239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:09.860451 (+ 212us) server_negotiation.cc:407] Connection header received
0807 06:27:09.860614 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:09.860617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:09.860673 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:09.860771 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:09.861123 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:09.861968 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:09.863139 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:09.863410 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:09.863531 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:09.864532 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:09.864537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:09.864539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:27:09.864555 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:09.864587 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:27:09.864603 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:09.864698 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:09.864937 (+ 239us) server_negotiation.cc:299] Negotiation successful
0807 06:27:09.864977 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260807 06:27:09.868588 14316 catalog_manager.cc:1347] Generated new cluster ID: 447781b33b2443f8b792df1f843eb998
I20260807 06:27:09.868659 14316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:27:09.883427 14316 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:27:09.884382 14316 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:27:09.893116 14316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a667659df8d4a449b7f8f37e07accd7: Generated new TSK 0
I20260807 06:27:09.893919 14316 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:27:09.899538 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 14199
W20260807 06:27:09.915441 14182 connection.cc:535] client connection to 127.0.0.1:40809 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40809): Connection reset by peer (error 104)
2026-08-07T06:27:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1667 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:27:09 dist-test-slave-44d2 krb5kdc[14342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:27:09 dist-test-slave-44d2 krb5kdc[14342](info): set up 2 sockets
Aug 07 06:27:09 dist-test-slave-44d2 krb5kdc[14342](info): commencing operation
krb5kdc: starting...
W20260807 06:27:11.134161 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.203s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:27:11 dist-test-slave-44d2 krb5kdc[14342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:27:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:27:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 06:27:11.275485 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45171
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:27:11.391499 14363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:27:11.391811 14363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:27:11.391893 14363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:27:11.396006 14363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:27:11.396111 14363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:27:11.396152 14363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:27:11.396210 14363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:27:11.396271 14363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:27:11.401551 14363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45171
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:27:11.402555 14363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:27:11.403550 14363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:27:11.412345 14374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:11.412375 14370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:11.412736 14369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:27:11.413026 14363 server_base.cc:1034] running on GCE node
I20260807 06:27:11.413581 14363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:27:11.414654 14363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:27:11.415874 14363 hybrid_clock.cc:648] HybridClock initialized: now 1786084031415842 us; error 43 us; skew 500 ppm
Aug 07 06:27:11 dist-test-slave-44d2 krb5kdc[14342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:27:11.418956 14363 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 06:27:11.419860 14363 webserver.cc:460] Webserver started at http://127.0.0.1:40463/ using document root <none> and password file <none>
I20260807 06:27:11.420444 14363 fs_manager.cc:359] Metadata directory not provided
I20260807 06:27:11.420565 14363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:27:11.420791 14363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:27:11.423627 14363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "4ce626c6e9e9484fbfdb041b231dc346"
format_stamp: "Formatted at 2026-08-07 06:27:11 on dist-test-slave-44d2"
server_key: "aed845eaa597c7e361b513ef18fb1329"
server_key_iv: "3ce087fa2fb347a9968102563c070657"
server_key_version: "encryptionkey@0"
I20260807 06:27:11.424381 14363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "4ce626c6e9e9484fbfdb041b231dc346"
format_stamp: "Formatted at 2026-08-07 06:27:11 on dist-test-slave-44d2"
server_key: "aed845eaa597c7e361b513ef18fb1329"
server_key_iv: "3ce087fa2fb347a9968102563c070657"
server_key_version: "encryptionkey@0"
I20260807 06:27:11.430158 14363 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 06:27:11.433458 14381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:27:11.435274 14363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:27:11.435473 14363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "4ce626c6e9e9484fbfdb041b231dc346"
format_stamp: "Formatted at 2026-08-07 06:27:11 on dist-test-slave-44d2"
server_key: "aed845eaa597c7e361b513ef18fb1329"
server_key_iv: "3ce087fa2fb347a9968102563c070657"
server_key_version: "encryptionkey@0"
I20260807 06:27:11.435616 14363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:27:11.444747 14363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:27:11.448834 14363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:27:11.449051 14363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:27:11.462347 14363 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45171
I20260807 06:27:11.462368 14454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45171 every 8 connection(s)
I20260807 06:27:11.463580 14363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:27:11.465651 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 14363
I20260807 06:27:11.465806 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:27:11.466142 16376 external_mini_cluster.cc:1428] Setting key 84f26fc08fbdedc94b9f39c532d13903
I20260807 06:27:11.469076 14455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:27:11.472241 14458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:11.467894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:58063 (local address 127.0.0.1:45171)
0807 06:27:11.468517 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:11.468541 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:11.468587 (+ 46us) server_negotiation.cc:407] Connection header received
0807 06:27:11.469394 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:11.469415 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:11.469717 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:11.470093 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:11.470881 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:11.470887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:11.470901 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:27:11.470911 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:11.470963 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:27:11.470989 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:11.471152 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:11.471225 (+ 73us) server_negotiation.cc:299] Negotiation successful
0807 06:27:11.471532 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":173,"threads_started":1}
I20260807 06:27:11.475314 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:27:11.477771 14455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ce626c6e9e9484fbfdb041b231dc346: Bootstrap starting.
I20260807 06:27:11.480943 14455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ce626c6e9e9484fbfdb041b231dc346: Neither blocks nor log segments found. Creating new log.
I20260807 06:27:11.481545 14458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:11.479429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:47023 (local address 127.0.0.1:45171)
0807 06:27:11.479570 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:11.479575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:11.480034 (+ 459us) server_negotiation.cc:407] Connection header received
0807 06:27:11.480134 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:11.480136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:11.480194 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:11.480280 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:11.481151 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:11.481157 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:11.481161 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:27:11.481174 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:11.481204 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:27:11.481217 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:11.481297 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:11.481374 (+ 77us) server_negotiation.cc:299] Negotiation successful
0807 06:27:11.481419 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 06:27:11.482203 14455 log.cc:824] T 00000000000000000000000000000000 P 4ce626c6e9e9484fbfdb041b231dc346: Log is configured to *not* fsync() on all Append() calls
I20260807 06:27:11.484822 14455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ce626c6e9e9484fbfdb041b231dc346: No bootstrap required, opened a new log
I20260807 06:27:11.490883 14455 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ce626c6e9e9484fbfdb041b231dc346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce626c6e9e9484fbfdb041b231dc346" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45171 } }
I20260807 06:27:11.491168 14455 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ce626c6e9e9484fbfdb041b231dc346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:27:11.491209 14455 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ce626c6e9e9484fbfdb041b231dc346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ce626c6e9e9484fbfdb041b231dc346, State: Initialized, Role: FOLLOWER
I20260807 06:27:11.491587 14455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ce626c6e9e9484fbfdb041b231dc346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce626c6e9e9484fbfdb041b231dc346" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45171 } }
I20260807 06:27:11.491703 14455 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ce626c6e9e9484fbfdb041b231dc346 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:27:11.491748 14455 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ce626c6e9e9484fbfdb041b231dc346 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:27:11.491822 14455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ce626c6e9e9484fbfdb041b231dc346 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:27:11.492794 14455 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ce626c6e9e9484fbfdb041b231dc346 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce626c6e9e9484fbfdb041b231dc346" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45171 } }
I20260807 06:27:11.493106 14455 leader_election.cc:302] T 00000000000000000000000000000000 P 4ce626c6e9e9484fbfdb041b231dc346 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ce626c6e9e9484fbfdb041b231dc346; no voters:
I20260807 06:27:11.493373 14455 leader_election.cc:288] T 00000000000000000000000000000000 P 4ce626c6e9e9484fbfdb041b231dc346 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:27:11.493553 14471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ce626c6e9e9484fbfdb041b231dc346 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:27:11.493825 14471 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ce626c6e9e9484fbfdb041b231dc346 [term 1 LEADER]: Becoming Leader. State: Replica: 4ce626c6e9e9484fbfdb041b231dc346, State: Running, Role: LEADER
I20260807 06:27:11.494158 14471 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ce626c6e9e9484fbfdb041b231dc346 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce626c6e9e9484fbfdb041b231dc346" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45171 } }
I20260807 06:27:11.494825 14455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ce626c6e9e9484fbfdb041b231dc346 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:27:11.495855 14473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce626c6e9e9484fbfdb041b231dc346 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ce626c6e9e9484fbfdb041b231dc346. Latest consensus state: current_term: 1 leader_uuid: "4ce626c6e9e9484fbfdb041b231dc346" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce626c6e9e9484fbfdb041b231dc346" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45171 } } }
I20260807 06:27:11.495985 14473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce626c6e9e9484fbfdb041b231dc346 [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:11.496305 14472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce626c6e9e9484fbfdb041b231dc346 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ce626c6e9e9484fbfdb041b231dc346" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce626c6e9e9484fbfdb041b231dc346" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45171 } } }
I20260807 06:27:11.496416 14472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce626c6e9e9484fbfdb041b231dc346 [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:11.497645 14486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:27:11.500401 14486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:27:11.506037 14486 catalog_manager.cc:1347] Generated new cluster ID: 569364398b4542fba52f745a4fa9454d
I20260807 06:27:11.506130 14486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:27:11.537315 14486 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:27:11.537693 14486 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:27:11.545439 14486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ce626c6e9e9484fbfdb041b231dc346: Generated new TSK 0
I20260807 06:27:11.546197 14486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:27:11.555748 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 14363
2026-08-07T06:27:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1652 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:27:11 dist-test-slave-44d2 krb5kdc[14502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:27:11 dist-test-slave-44d2 krb5kdc[14502](info): set up 2 sockets
Aug 07 06:27:11 dist-test-slave-44d2 krb5kdc[14502](info): commencing operation
krb5kdc: starting...
W20260807 06:27:12.816208 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.233s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:27:12 dist-test-slave-44d2 krb5kdc[14502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:27:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:27:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:27:12.951807 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:42253
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:35709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:42253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:27:13.068936 14520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:27:13.069263 14520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:27:13.069371 14520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:27:13.073405 14520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:27:13.073505 14520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:27:13.073556 14520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:27:13.073623 14520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:27:13.073668 14520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:27:13.078966 14520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:35709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:42253
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:42253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:27:13.080083 14520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:27:13.081204 14520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:27:13.090186 14530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:13.090214 14534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:27:13.090492 14520 server_base.cc:1034] running on GCE node
W20260807 06:27:13.090415 14531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:27:13.091264 14520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:27:13.092396 14520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:27:13.093575 14520 hybrid_clock.cc:648] HybridClock initialized: now 1786084033093560 us; error 27 us; skew 500 ppm
Aug 07 06:27:13 dist-test-slave-44d2 krb5kdc[14502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084033, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:27:13.096567 14520 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:27:13.097412 14520 webserver.cc:460] Webserver started at http://127.15.254.62:36337/ using document root <none> and password file <none>
I20260807 06:27:13.097970 14520 fs_manager.cc:359] Metadata directory not provided
I20260807 06:27:13.098034 14520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:27:13.098204 14520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:27:13.100263 14520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "81290f8f7bea43b8b009d75571be798d"
format_stamp: "Formatted at 2026-08-07 06:27:13 on dist-test-slave-44d2"
server_key: "a79b81cd0c2936546a55e9b608a89dfb"
server_key_iv: "315d328e00a26d8b5e442fd4e77fa9a4"
server_key_version: "encryptionkey@0"
I20260807 06:27:13.100859 14520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "81290f8f7bea43b8b009d75571be798d"
format_stamp: "Formatted at 2026-08-07 06:27:13 on dist-test-slave-44d2"
server_key: "a79b81cd0c2936546a55e9b608a89dfb"
server_key_iv: "315d328e00a26d8b5e442fd4e77fa9a4"
server_key_version: "encryptionkey@0"
I20260807 06:27:13.104458 14520 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:27:13.106868 14548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:27:13.108137 14520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 06:27:13.108283 14520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "81290f8f7bea43b8b009d75571be798d"
format_stamp: "Formatted at 2026-08-07 06:27:13 on dist-test-slave-44d2"
server_key: "a79b81cd0c2936546a55e9b608a89dfb"
server_key_iv: "315d328e00a26d8b5e442fd4e77fa9a4"
server_key_version: "encryptionkey@0"
I20260807 06:27:13.108388 14520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:27:13.131127 14520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:27:13.134575 14520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:27:13.134799 14520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:27:13.142227 14520 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:42253
I20260807 06:27:13.142270 14616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:42253 every 8 connection(s)
I20260807 06:27:13.143242 14520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:27:13.146267 14618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:27:13.149703 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 14520
I20260807 06:27:13.149969 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:27:13.150326 16376 external_mini_cluster.cc:1428] Setting key 8db1abe726031c7e407fc39c2282b7d1
I20260807 06:27:13.152756 14618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81290f8f7bea43b8b009d75571be798d: Bootstrap starting.
I20260807 06:27:13.155118 14618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81290f8f7bea43b8b009d75571be798d: Neither blocks nor log segments found. Creating new log.
I20260807 06:27:13.156139 14618 log.cc:824] T 00000000000000000000000000000000 P 81290f8f7bea43b8b009d75571be798d: Log is configured to *not* fsync() on all Append() calls
I20260807 06:27:13.158371 14618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81290f8f7bea43b8b009d75571be798d: No bootstrap required, opened a new log
Aug 07 06:27:13 dist-test-slave-44d2 krb5kdc[14502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:27:13.162608 14618 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81290f8f7bea43b8b009d75571be798d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81290f8f7bea43b8b009d75571be798d" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 42253 } }
I20260807 06:27:13.162930 14618 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81290f8f7bea43b8b009d75571be798d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:27:13.162994 14618 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81290f8f7bea43b8b009d75571be798d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81290f8f7bea43b8b009d75571be798d, State: Initialized, Role: FOLLOWER
I20260807 06:27:13.163525 14618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81290f8f7bea43b8b009d75571be798d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81290f8f7bea43b8b009d75571be798d" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 42253 } }
I20260807 06:27:13.163712 14618 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81290f8f7bea43b8b009d75571be798d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:27:13.163779 14618 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81290f8f7bea43b8b009d75571be798d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:27:13.163887 14618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81290f8f7bea43b8b009d75571be798d [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:27:13.165155 14618 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81290f8f7bea43b8b009d75571be798d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81290f8f7bea43b8b009d75571be798d" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 42253 } }
I20260807 06:27:13.165550 14618 leader_election.cc:302] T 00000000000000000000000000000000 P 81290f8f7bea43b8b009d75571be798d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81290f8f7bea43b8b009d75571be798d; no voters:
I20260807 06:27:13.165984 14618 leader_election.cc:288] T 00000000000000000000000000000000 P 81290f8f7bea43b8b009d75571be798d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:27:13.166057 14625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81290f8f7bea43b8b009d75571be798d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:27:13.166280 14625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81290f8f7bea43b8b009d75571be798d [term 1 LEADER]: Becoming Leader. State: Replica: 81290f8f7bea43b8b009d75571be798d, State: Running, Role: LEADER
I20260807 06:27:13.166602 14625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81290f8f7bea43b8b009d75571be798d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81290f8f7bea43b8b009d75571be798d" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 42253 } }
I20260807 06:27:13.167448 14618 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81290f8f7bea43b8b009d75571be798d [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:27:13.167520 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:13.151985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55059 (local address 127.15.254.62:42253)
0807 06:27:13.152609 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:13.152629 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:13.152691 (+ 62us) server_negotiation.cc:407] Connection header received
0807 06:27:13.153527 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:13.153547 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:13.153926 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:13.154308 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:13.155422 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:13.156865 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:13.157872 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:13.158562 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:13.158722 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:13.161670 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:13.161677 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:13.161698 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:13.161737 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:13.163849 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:13.164452 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:13.164459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:13.164465 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:13.164632 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:13.164997 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:13.165004 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:13.165009 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:13.165592 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:13.165810 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:13.166395 (+ 585us) server_negotiation.cc:299] Negotiation successful
0807 06:27:13.166675 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":149,"threads_started":1}
I20260807 06:27:13.167898 14625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81290f8f7bea43b8b009d75571be798d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81290f8f7bea43b8b009d75571be798d. Latest consensus state: current_term: 1 leader_uuid: "81290f8f7bea43b8b009d75571be798d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81290f8f7bea43b8b009d75571be798d" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 42253 } } }
I20260807 06:27:13.168016 14625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81290f8f7bea43b8b009d75571be798d [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:13.168219 14625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81290f8f7bea43b8b009d75571be798d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81290f8f7bea43b8b009d75571be798d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81290f8f7bea43b8b009d75571be798d" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 42253 } } }
I20260807 06:27:13.168308 14625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81290f8f7bea43b8b009d75571be798d [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:13.168891 14630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:27:13.171572 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:27:13.172322 14630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:27:13.178820 14630 catalog_manager.cc:1347] Generated new cluster ID: c050a6d82e9f483b8e2b1466272d6680
I20260807 06:27:13.178913 14630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:27:13.184979 14630 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:27:13.185316 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:13.175082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53305 (local address 127.15.254.62:42253)
0807 06:27:13.175225 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:13.175230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:13.175817 (+ 587us) server_negotiation.cc:407] Connection header received
0807 06:27:13.175963 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:13.175965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:13.176015 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:13.176105 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:13.177078 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:13.177663 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:13.178582 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:13.178976 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:13.179079 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:13.181045 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:13.181049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:13.181052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:13.181078 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:13.182913 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:13.183559 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:13.183562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:13.183564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:13.183651 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:13.184017 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:13.184021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:13.184022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:13.184228 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:13.184374 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:13.185079 (+ 705us) server_negotiation.cc:299] Negotiation successful
0807 06:27:13.185172 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260807 06:27:13.186201 14630 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:27:13.190240 14630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81290f8f7bea43b8b009d75571be798d: Generated new TSK 0
I20260807 06:27:13.191048 14630 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:27:13.198814 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 14520
W20260807 06:27:13.215003 14500 connection.cc:535] client connection to 127.15.254.62:42253 recv error: Network error: failed to read from TLS socket (remote: 127.15.254.62:42253): Connection reset by peer (error 104)
2026-08-07T06:27:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1646 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:27:13 dist-test-slave-44d2 krb5kdc[14659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:27:13 dist-test-slave-44d2 krb5kdc[14659](info): set up 2 sockets
Aug 07 06:27:13 dist-test-slave-44d2 krb5kdc[14659](info): commencing operation
krb5kdc: starting...
W20260807 06:27:14.514478 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.285s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:27:14 dist-test-slave-44d2 krb5kdc[14659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:27:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:27:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:27:14.652930 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:43981
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:43981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:27:14.770742 14678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:27:14.771046 14678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:27:14.771126 14678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:27:14.775173 14678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:27:14.775264 14678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:27:14.775303 14678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:27:14.775360 14678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:27:14.775406 14678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:27:14.780654 14678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:43981
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:43981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:27:14.781746 14678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:27:14.782824 14678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:27:14.791981 14683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:27:14.792164 14678 server_base.cc:1034] running on GCE node
W20260807 06:27:14.792044 14687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:14.792044 14684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:27:14.792969 14678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:27:14.794169 14678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:27:14.795380 14678 hybrid_clock.cc:648] HybridClock initialized: now 1786084034795362 us; error 31 us; skew 500 ppm
Aug 07 06:27:14 dist-test-slave-44d2 krb5kdc[14659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084034, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:27:14.798286 14678 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:27:14.799214 14678 webserver.cc:460] Webserver started at http://127.15.254.62:39795/ using document root <none> and password file <none>
I20260807 06:27:14.799857 14678 fs_manager.cc:359] Metadata directory not provided
I20260807 06:27:14.799955 14678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:27:14.800197 14678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:27:14.802326 14678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "5547cee585f3464b979d5ba2c15aba07"
format_stamp: "Formatted at 2026-08-07 06:27:14 on dist-test-slave-44d2"
server_key: "d6e5ae21cb67dbb3fd31c6bb421f1405"
server_key_iv: "508412f8728fb22e714f17fd3b148aaa"
server_key_version: "encryptionkey@0"
I20260807 06:27:14.802938 14678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "5547cee585f3464b979d5ba2c15aba07"
format_stamp: "Formatted at 2026-08-07 06:27:14 on dist-test-slave-44d2"
server_key: "d6e5ae21cb67dbb3fd31c6bb421f1405"
server_key_iv: "508412f8728fb22e714f17fd3b148aaa"
server_key_version: "encryptionkey@0"
I20260807 06:27:14.808243 14678 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260807 06:27:14.811009 14702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:27:14.812597 14678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:27:14.812741 14678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "5547cee585f3464b979d5ba2c15aba07"
format_stamp: "Formatted at 2026-08-07 06:27:14 on dist-test-slave-44d2"
server_key: "d6e5ae21cb67dbb3fd31c6bb421f1405"
server_key_iv: "508412f8728fb22e714f17fd3b148aaa"
server_key_version: "encryptionkey@0"
I20260807 06:27:14.812858 14678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:27:14.831566 14678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:27:14.834964 14678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:27:14.835175 14678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:27:14.842260 14678 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:43981
I20260807 06:27:14.842326 14773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:43981 every 8 connection(s)
I20260807 06:27:14.843274 14678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:27:14.846154 14774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:27:14.850796 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 14678
I20260807 06:27:14.850973 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:27:14.851306 16376 external_mini_cluster.cc:1428] Setting key fccf840be14df199d71bec9168353e2f
I20260807 06:27:14.852116 14774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5547cee585f3464b979d5ba2c15aba07: Bootstrap starting.
I20260807 06:27:14.854707 14774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5547cee585f3464b979d5ba2c15aba07: Neither blocks nor log segments found. Creating new log.
I20260807 06:27:14.855480 14774 log.cc:824] T 00000000000000000000000000000000 P 5547cee585f3464b979d5ba2c15aba07: Log is configured to *not* fsync() on all Append() calls
I20260807 06:27:14.857496 14774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5547cee585f3464b979d5ba2c15aba07: No bootstrap required, opened a new log
I20260807 06:27:14.860800 14778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:14.852849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50889 (local address 127.15.254.62:43981)
0807 06:27:14.853339 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:14.853354 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:14.853399 (+ 45us) server_negotiation.cc:407] Connection header received
0807 06:27:14.854056 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:14.854075 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:14.854491 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:14.854860 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:14.855286 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:14.856343 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:14.857305 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:14.857836 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:14.858019 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:14.858978 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:14.858984 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:14.858997 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:27:14.859418 (+ 421us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:14.859472 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:27:14.859496 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:14.859642 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:14.859833 (+ 191us) server_negotiation.cc:299] Negotiation successful
0807 06:27:14.859981 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":138,"threads_started":1}
I20260807 06:27:14.861231 14774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5547cee585f3464b979d5ba2c15aba07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5547cee585f3464b979d5ba2c15aba07" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43981 } }
I20260807 06:27:14.861485 14774 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5547cee585f3464b979d5ba2c15aba07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:27:14.861557 14774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5547cee585f3464b979d5ba2c15aba07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5547cee585f3464b979d5ba2c15aba07, State: Initialized, Role: FOLLOWER
I20260807 06:27:14.861979 14774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5547cee585f3464b979d5ba2c15aba07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5547cee585f3464b979d5ba2c15aba07" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43981 } }
I20260807 06:27:14.862133 14774 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5547cee585f3464b979d5ba2c15aba07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:27:14.862215 14774 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5547cee585f3464b979d5ba2c15aba07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:27:14.862336 14774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5547cee585f3464b979d5ba2c15aba07 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:27:14.863440 14774 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5547cee585f3464b979d5ba2c15aba07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5547cee585f3464b979d5ba2c15aba07" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43981 } }
I20260807 06:27:14.863780 14774 leader_election.cc:302] T 00000000000000000000000000000000 P 5547cee585f3464b979d5ba2c15aba07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5547cee585f3464b979d5ba2c15aba07; no voters:
I20260807 06:27:14.863983 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:27:14.864154 14774 leader_election.cc:288] T 00000000000000000000000000000000 P 5547cee585f3464b979d5ba2c15aba07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:27:14.864249 14780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5547cee585f3464b979d5ba2c15aba07 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:27:14.864548 14780 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5547cee585f3464b979d5ba2c15aba07 [term 1 LEADER]: Becoming Leader. State: Replica: 5547cee585f3464b979d5ba2c15aba07, State: Running, Role: LEADER
I20260807 06:27:14.864867 14780 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5547cee585f3464b979d5ba2c15aba07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5547cee585f3464b979d5ba2c15aba07" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43981 } }
I20260807 06:27:14.865564 14774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5547cee585f3464b979d5ba2c15aba07 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:27:14.866150 14780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5547cee585f3464b979d5ba2c15aba07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5547cee585f3464b979d5ba2c15aba07. Latest consensus state: current_term: 1 leader_uuid: "5547cee585f3464b979d5ba2c15aba07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5547cee585f3464b979d5ba2c15aba07" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43981 } } }
I20260807 06:27:14.866279 14780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5547cee585f3464b979d5ba2c15aba07 [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:14.866606 14782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5547cee585f3464b979d5ba2c15aba07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5547cee585f3464b979d5ba2c15aba07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5547cee585f3464b979d5ba2c15aba07" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43981 } } }
I20260807 06:27:14.866699 14782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5547cee585f3464b979d5ba2c15aba07 [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:14.868026 14796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:27:14.871377 14778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:14.866580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53471 (local address 127.15.254.62:43981)
0807 06:27:14.866709 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:14.866713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:14.867008 (+ 295us) server_negotiation.cc:407] Connection header received
0807 06:27:14.867106 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:14.867109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:14.867147 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:14.867244 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:14.867526 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:14.868518 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:14.869610 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:14.869854 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:14.869964 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:14.870888 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:14.870891 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:14.870894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:27:14.870907 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:14.870941 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:27:14.870952 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:14.871052 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:14.871178 (+ 126us) server_negotiation.cc:299] Negotiation successful
0807 06:27:14.871211 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 06:27:14.871526 14796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:27:14.877558 14796 catalog_manager.cc:1347] Generated new cluster ID: 4ad2b3e74328429e83ea6663240614ce
I20260807 06:27:14.877625 14796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:27:14.892385 14796 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:27:14.893512 14796 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:27:14.900530 14796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5547cee585f3464b979d5ba2c15aba07: Generated new TSK 0
I20260807 06:27:14.901292 14796 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:27:14.913686 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 14678
W20260807 06:27:14.929327 14655 connection.cc:535] client connection to 127.15.254.62:43981 recv error: Network error: failed to read from TLS socket (remote: 127.15.254.62:43981): Connection reset by peer (error 104)
2026-08-07T06:27:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1714 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:27:14 dist-test-slave-44d2 krb5kdc[14814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:27:14 dist-test-slave-44d2 krb5kdc[14814](info): set up 2 sockets
Aug 07 06:27:14 dist-test-slave-44d2 krb5kdc[14814](info): commencing operation
krb5kdc: starting...
W20260807 06:27:16.161149 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.217s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:27:16 dist-test-slave-44d2 krb5kdc[14814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:27:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:27:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:27:16.301200 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:41317
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:41317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:27:16.416466 14829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:27:16.416740 14829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:27:16.416780 14829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:27:16.420706 14829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:27:16.420759 14829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:27:16.420778 14829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:27:16.420795 14829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:27:16.420810 14829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:27:16.425945 14829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:41317
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:41317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:27:16.426915 14829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:27:16.427877 14829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 06:27:16.436787 14829 server_base.cc:1034] running on GCE node
W20260807 06:27:16.436753 14835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:16.436754 14834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:16.437158 14837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:27:16.437736 14829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:27:16.438859 14829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:27:16.440099 14829 hybrid_clock.cc:648] HybridClock initialized: now 1786084036440064 us; error 47 us; skew 500 ppm
Aug 07 06:27:16 dist-test-slave-44d2 krb5kdc[14814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084036, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:27:16.443162 14829 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:27:16.444188 14829 webserver.cc:460] Webserver started at http://127.15.254.62:34627/ using document root <none> and password file <none>
I20260807 06:27:16.444859 14829 fs_manager.cc:359] Metadata directory not provided
I20260807 06:27:16.444954 14829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:27:16.445212 14829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:27:16.447256 14829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "14d72c6b36aa4a88a73955aaa6dfaa73"
format_stamp: "Formatted at 2026-08-07 06:27:16 on dist-test-slave-44d2"
server_key: "6a60fecdc317b2ea55efe2ee30970d05"
server_key_iv: "f46b3c62e93c9cc5d1b559ddd456cbfd"
server_key_version: "encryptionkey@0"
I20260807 06:27:16.447835 14829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "14d72c6b36aa4a88a73955aaa6dfaa73"
format_stamp: "Formatted at 2026-08-07 06:27:16 on dist-test-slave-44d2"
server_key: "6a60fecdc317b2ea55efe2ee30970d05"
server_key_iv: "f46b3c62e93c9cc5d1b559ddd456cbfd"
server_key_version: "encryptionkey@0"
I20260807 06:27:16.451632 14829 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 06:27:16.454382 14849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:27:16.455755 14829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 06:27:16.456028 14829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "14d72c6b36aa4a88a73955aaa6dfaa73"
format_stamp: "Formatted at 2026-08-07 06:27:16 on dist-test-slave-44d2"
server_key: "6a60fecdc317b2ea55efe2ee30970d05"
server_key_iv: "f46b3c62e93c9cc5d1b559ddd456cbfd"
server_key_version: "encryptionkey@0"
I20260807 06:27:16.456157 14829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:27:16.461357 14829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:27:16.464664 14829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:27:16.464886 14829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:27:16.472206 14829 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:41317
I20260807 06:27:16.472220 14922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:41317 every 8 connection(s)
I20260807 06:27:16.473256 14829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:27:16.476217 14924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:27:16.478999 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 14829
I20260807 06:27:16.479184 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:27:16.479503 16376 external_mini_cluster.cc:1428] Setting key 404ad4e7e93d98c07fc5c8c41abd272f
I20260807 06:27:16.482149 14924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d72c6b36aa4a88a73955aaa6dfaa73: Bootstrap starting.
I20260807 06:27:16.484838 14924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14d72c6b36aa4a88a73955aaa6dfaa73: Neither blocks nor log segments found. Creating new log.
I20260807 06:27:16.485153 14927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:16.480942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54965 (local address 127.15.254.62:41317)
0807 06:27:16.481387 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:16.481403 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:16.481439 (+ 36us) server_negotiation.cc:407] Connection header received
0807 06:27:16.482105 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:16.482123 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:16.482459 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:16.482745 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:16.483606 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:16.483611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:16.483622 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:27:16.483633 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:16.483689 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:27:16.483709 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:16.483831 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:16.483953 (+ 122us) server_negotiation.cc:299] Negotiation successful
0807 06:27:16.484239 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":137,"threads_started":1}
I20260807 06:27:16.485777 14924 log.cc:824] T 00000000000000000000000000000000 P 14d72c6b36aa4a88a73955aaa6dfaa73: Log is configured to *not* fsync() on all Append() calls
I20260807 06:27:16.487982 14924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d72c6b36aa4a88a73955aaa6dfaa73: No bootstrap required, opened a new log
I20260807 06:27:16.488314 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:27:16.492044 14924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14d72c6b36aa4a88a73955aaa6dfaa73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d72c6b36aa4a88a73955aaa6dfaa73" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41317 } }
I20260807 06:27:16.492336 14924 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14d72c6b36aa4a88a73955aaa6dfaa73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:27:16.492345 14927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:16.490583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58543 (local address 127.15.254.62:41317)
0807 06:27:16.490699 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:16.490703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:16.490942 (+ 239us) server_negotiation.cc:407] Connection header received
0807 06:27:16.491036 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:16.491038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:16.491067 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:16.491132 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:16.491851 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:16.491854 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:16.491856 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:27:16.491866 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:16.491941 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:27:16.491958 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:16.492030 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:16.492163 (+ 133us) server_negotiation.cc:299] Negotiation successful
0807 06:27:16.492195 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 06:27:16.492419 14924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14d72c6b36aa4a88a73955aaa6dfaa73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d72c6b36aa4a88a73955aaa6dfaa73, State: Initialized, Role: FOLLOWER
I20260807 06:27:16.492919 14924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14d72c6b36aa4a88a73955aaa6dfaa73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d72c6b36aa4a88a73955aaa6dfaa73" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41317 } }
I20260807 06:27:16.493083 14924 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14d72c6b36aa4a88a73955aaa6dfaa73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:27:16.493158 14924 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14d72c6b36aa4a88a73955aaa6dfaa73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:27:16.493259 14924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14d72c6b36aa4a88a73955aaa6dfaa73 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:27:16.494134 14924 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14d72c6b36aa4a88a73955aaa6dfaa73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d72c6b36aa4a88a73955aaa6dfaa73" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41317 } }
I20260807 06:27:16.494422 14924 leader_election.cc:302] T 00000000000000000000000000000000 P 14d72c6b36aa4a88a73955aaa6dfaa73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14d72c6b36aa4a88a73955aaa6dfaa73; no voters:
I20260807 06:27:16.494661 14924 leader_election.cc:288] T 00000000000000000000000000000000 P 14d72c6b36aa4a88a73955aaa6dfaa73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:27:16.494818 14934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14d72c6b36aa4a88a73955aaa6dfaa73 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:27:16.495064 14934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14d72c6b36aa4a88a73955aaa6dfaa73 [term 1 LEADER]: Becoming Leader. State: Replica: 14d72c6b36aa4a88a73955aaa6dfaa73, State: Running, Role: LEADER
I20260807 06:27:16.495362 14934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14d72c6b36aa4a88a73955aaa6dfaa73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d72c6b36aa4a88a73955aaa6dfaa73" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41317 } }
I20260807 06:27:16.495882 14924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14d72c6b36aa4a88a73955aaa6dfaa73 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:27:16.497180 14938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d72c6b36aa4a88a73955aaa6dfaa73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14d72c6b36aa4a88a73955aaa6dfaa73. Latest consensus state: current_term: 1 leader_uuid: "14d72c6b36aa4a88a73955aaa6dfaa73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d72c6b36aa4a88a73955aaa6dfaa73" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41317 } } }
I20260807 06:27:16.497213 14936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d72c6b36aa4a88a73955aaa6dfaa73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14d72c6b36aa4a88a73955aaa6dfaa73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d72c6b36aa4a88a73955aaa6dfaa73" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41317 } } }
I20260807 06:27:16.497331 14938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d72c6b36aa4a88a73955aaa6dfaa73 [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:16.497335 14936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d72c6b36aa4a88a73955aaa6dfaa73 [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:16.497722 14950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:27:16.500819 14950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:27:16.506968 14950 catalog_manager.cc:1347] Generated new cluster ID: a54c333c96e94839ab3d5d1e37e259aa
I20260807 06:27:16.507045 14950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:27:16.518553 14950 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:27:16.518960 14950 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:27:16.524313 14950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14d72c6b36aa4a88a73955aaa6dfaa73: Generated new TSK 0
I20260807 06:27:16.525087 14950 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:27:16.533128 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 14829
2026-08-07T06:27:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1627 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:27:16 dist-test-slave-44d2 krb5kdc[14967](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:27:16 dist-test-slave-44d2 krb5kdc[14967](info): set up 2 sockets
Aug 07 06:27:16 dist-test-slave-44d2 krb5kdc[14967](info): commencing operation
krb5kdc: starting...
W20260807 06:27:17.815222 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.244s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:27:17 dist-test-slave-44d2 krb5kdc[14967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:27:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:27:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:27:17.956310 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:36385
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:43863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:36385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:27:18.074574 14984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:27:18.074837 14984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:27:18.074887 14984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:27:18.078918 14984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:27:18.078977 14984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:27:18.078996 14984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:27:18.079013 14984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:27:18.079028 14984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:27:18.084190 14984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:43863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:36385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:36385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:27:18.085297 14984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:27:18.086308 14984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:27:18.095224 14990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:18.095229 14996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:18.095229 14991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:27:18.095448 14984 server_base.cc:1034] running on GCE node
I20260807 06:27:18.096171 14984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:27:18.097425 14984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:27:18.098655 14984 hybrid_clock.cc:648] HybridClock initialized: now 1786084038098622 us; error 43 us; skew 500 ppm
Aug 07 06:27:18 dist-test-slave-44d2 krb5kdc[14967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084038, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:27:18.107611 14984 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:27:18.108752 14984 webserver.cc:460] Webserver started at http://127.15.254.62:34593/ using document root <none> and password file <none>
I20260807 06:27:18.109349 14984 fs_manager.cc:359] Metadata directory not provided
I20260807 06:27:18.109459 14984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:27:18.109707 14984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:27:18.111806 14984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "3db1cee7990d4f639dd35f4675229c5a"
format_stamp: "Formatted at 2026-08-07 06:27:18 on dist-test-slave-44d2"
server_key: "1e68fc603f9d9d667be91586b07bcdbb"
server_key_iv: "80c5b681986e0a56fcc5b372128d1ba4"
server_key_version: "encryptionkey@0"
I20260807 06:27:18.112399 14984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "3db1cee7990d4f639dd35f4675229c5a"
format_stamp: "Formatted at 2026-08-07 06:27:18 on dist-test-slave-44d2"
server_key: "1e68fc603f9d9d667be91586b07bcdbb"
server_key_iv: "80c5b681986e0a56fcc5b372128d1ba4"
server_key_version: "encryptionkey@0"
I20260807 06:27:18.116251 14984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 06:27:18.118952 15009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:27:18.120162 14984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 06:27:18.120326 14984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "3db1cee7990d4f639dd35f4675229c5a"
format_stamp: "Formatted at 2026-08-07 06:27:18 on dist-test-slave-44d2"
server_key: "1e68fc603f9d9d667be91586b07bcdbb"
server_key_iv: "80c5b681986e0a56fcc5b372128d1ba4"
server_key_version: "encryptionkey@0"
I20260807 06:27:18.120455 14984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:27:18.131989 14984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:27:18.135254 14984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:27:18.135476 14984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:27:18.142819 14984 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:36385
I20260807 06:27:18.142887 15085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:36385 every 8 connection(s)
I20260807 06:27:18.143771 14984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:27:18.145226 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 14984
I20260807 06:27:18.145385 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:27:18.145728 16376 external_mini_cluster.cc:1428] Setting key 3442d64a15b7b74c51c33fac9a51e791
I20260807 06:27:18.147055 15087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:27:18.152709 15087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3db1cee7990d4f639dd35f4675229c5a: Bootstrap starting.
Aug 07 06:27:18 dist-test-slave-44d2 krb5kdc[14967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:27:18.155030 15087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3db1cee7990d4f639dd35f4675229c5a: Neither blocks nor log segments found. Creating new log.
I20260807 06:27:18.155769 15087 log.cc:824] T 00000000000000000000000000000000 P 3db1cee7990d4f639dd35f4675229c5a: Log is configured to *not* fsync() on all Append() calls
I20260807 06:27:18.157676 15087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3db1cee7990d4f639dd35f4675229c5a: No bootstrap required, opened a new log
I20260807 06:27:18.159444 15092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:18.147154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39077 (local address 127.15.254.62:36385)
0807 06:27:18.147590 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:18.147605 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:18.147640 (+ 35us) server_negotiation.cc:407] Connection header received
0807 06:27:18.148290 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:18.148307 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:18.148676 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:18.149009 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:18.149813 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:18.150793 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:18.151663 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:18.152161 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:18.152329 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:18.154842 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:18.154847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:18.154862 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:18.154889 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:18.156656 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:18.157100 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:18.157110 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:18.157115 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:18.157220 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:18.157502 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:18.157507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:18.157508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:18.157952 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:18.158148 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:18.158457 (+ 309us) server_negotiation.cc:299] Negotiation successful
0807 06:27:18.158746 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":147,"threads_started":1}
I20260807 06:27:18.161669 15087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3db1cee7990d4f639dd35f4675229c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db1cee7990d4f639dd35f4675229c5a" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36385 } }
I20260807 06:27:18.161988 15087 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3db1cee7990d4f639dd35f4675229c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:27:18.162068 15087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3db1cee7990d4f639dd35f4675229c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db1cee7990d4f639dd35f4675229c5a, State: Initialized, Role: FOLLOWER
I20260807 06:27:18.162503 15087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3db1cee7990d4f639dd35f4675229c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db1cee7990d4f639dd35f4675229c5a" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36385 } }
I20260807 06:27:18.162683 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:27:18.162688 15087 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3db1cee7990d4f639dd35f4675229c5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:27:18.162778 15087 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3db1cee7990d4f639dd35f4675229c5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:27:18.162880 15087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3db1cee7990d4f639dd35f4675229c5a [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:27:18.163825 15087 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3db1cee7990d4f639dd35f4675229c5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db1cee7990d4f639dd35f4675229c5a" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36385 } }
I20260807 06:27:18.164213 15087 leader_election.cc:302] T 00000000000000000000000000000000 P 3db1cee7990d4f639dd35f4675229c5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3db1cee7990d4f639dd35f4675229c5a; no voters:
I20260807 06:27:18.164633 15087 leader_election.cc:288] T 00000000000000000000000000000000 P 3db1cee7990d4f639dd35f4675229c5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:27:18.164742 15100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3db1cee7990d4f639dd35f4675229c5a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:27:18.165015 15100 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3db1cee7990d4f639dd35f4675229c5a [term 1 LEADER]: Becoming Leader. State: Replica: 3db1cee7990d4f639dd35f4675229c5a, State: Running, Role: LEADER
I20260807 06:27:18.165324 15100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3db1cee7990d4f639dd35f4675229c5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db1cee7990d4f639dd35f4675229c5a" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36385 } }
I20260807 06:27:18.165984 15087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3db1cee7990d4f639dd35f4675229c5a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:27:18.166589 15100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3db1cee7990d4f639dd35f4675229c5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3db1cee7990d4f639dd35f4675229c5a. Latest consensus state: current_term: 1 leader_uuid: "3db1cee7990d4f639dd35f4675229c5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db1cee7990d4f639dd35f4675229c5a" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36385 } } }
I20260807 06:27:18.166720 15100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3db1cee7990d4f639dd35f4675229c5a [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:18.167204 15102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3db1cee7990d4f639dd35f4675229c5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3db1cee7990d4f639dd35f4675229c5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db1cee7990d4f639dd35f4675229c5a" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36385 } } }
I20260807 06:27:18.167316 15102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3db1cee7990d4f639dd35f4675229c5a [sys.catalog]: This master's current role is: LEADER
W20260807 06:27:18.171435 15121 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3db1cee7990d4f639dd35f4675229c5a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:27:18.171514 15121 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:27:18.171793 15110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:27:18.173075 15110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:27:18.174561 15092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:18.165131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34037 (local address 127.15.254.62:36385)
0807 06:27:18.165235 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:18.165239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:18.165397 (+ 158us) server_negotiation.cc:407] Connection header received
0807 06:27:18.165580 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:18.165583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:18.165632 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:18.165710 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:18.166639 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:18.167534 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:18.168957 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:18.169263 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:18.169337 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:18.171185 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:18.171189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:18.171192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:18.171216 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:18.172816 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:18.173345 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:18.173356 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:18.173358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:18.173440 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:18.173727 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:18.173731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:18.173733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:18.173908 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:18.174042 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:18.174276 (+ 234us) server_negotiation.cc:299] Negotiation successful
0807 06:27:18.174401 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260807 06:27:18.179294 15110 catalog_manager.cc:1347] Generated new cluster ID: 3485bb6044d04319b47ddf7d3665da7e
I20260807 06:27:18.179373 15110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:27:18.187408 15110 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:27:18.188354 15110 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:27:18.193885 15110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3db1cee7990d4f639dd35f4675229c5a: Generated new TSK 0
I20260807 06:27:18.194515 15110 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:27:18.200512 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 14984
2026-08-07T06:27:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1657 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:27:18 dist-test-slave-44d2 krb5kdc[15127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:27:18 dist-test-slave-44d2 krb5kdc[15127](info): set up 2 sockets
Aug 07 06:27:18 dist-test-slave-44d2 krb5kdc[15127](info): commencing operation
krb5kdc: starting...
W20260807 06:27:19.459290 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.230s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:27:19 dist-test-slave-44d2 krb5kdc[15127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:27:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:27:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:27:19.600101 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:44123
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:40019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:44123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:27:19.715617 15149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:27:19.715960 15149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:27:19.716038 15149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:27:19.720106 15149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:27:19.720216 15149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:27:19.720255 15149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:27:19.720312 15149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:27:19.720350 15149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:27:19.726240 15149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:40019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:44123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:44123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:27:19.727298 15149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:27:19.728286 15149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:27:19.737289 15157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:19.737418 15154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:19.737285 15155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:27:19.737663 15149 server_base.cc:1034] running on GCE node
I20260807 06:27:19.738231 15149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:27:19.739440 15149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:27:19.740643 15149 hybrid_clock.cc:648] HybridClock initialized: now 1786084039740610 us; error 42 us; skew 500 ppm
Aug 07 06:27:19 dist-test-slave-44d2 krb5kdc[15127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084039, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:27:19.743700 15149 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:27:19.744666 15149 webserver.cc:460] Webserver started at http://127.15.254.62:37893/ using document root <none> and password file <none>
I20260807 06:27:19.745204 15149 fs_manager.cc:359] Metadata directory not provided
I20260807 06:27:19.745263 15149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:27:19.745446 15149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:27:19.747524 15149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "8a1f34ea29c14c00b94cc7dbf024d4cc"
format_stamp: "Formatted at 2026-08-07 06:27:19 on dist-test-slave-44d2"
server_key: "1d2613c1f728551229da58ddf38af4cc"
server_key_iv: "3145bcf0a980a62194d2cffa9444d04c"
server_key_version: "encryptionkey@0"
I20260807 06:27:19.748054 15149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "8a1f34ea29c14c00b94cc7dbf024d4cc"
format_stamp: "Formatted at 2026-08-07 06:27:19 on dist-test-slave-44d2"
server_key: "1d2613c1f728551229da58ddf38af4cc"
server_key_iv: "3145bcf0a980a62194d2cffa9444d04c"
server_key_version: "encryptionkey@0"
I20260807 06:27:19.751964 15149 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 06:27:19.754499 15166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:27:19.756002 15149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:27:19.756176 15149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "8a1f34ea29c14c00b94cc7dbf024d4cc"
format_stamp: "Formatted at 2026-08-07 06:27:19 on dist-test-slave-44d2"
server_key: "1d2613c1f728551229da58ddf38af4cc"
server_key_iv: "3145bcf0a980a62194d2cffa9444d04c"
server_key_version: "encryptionkey@0"
I20260807 06:27:19.756289 15149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:27:19.778952 15149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:27:19.782799 15149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:27:19.783031 15149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:27:19.791260 15149 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:44123
I20260807 06:27:19.791287 15237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:44123 every 8 connection(s)
I20260807 06:27:19.792385 15149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:27:19.795490 15238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:27:19.798189 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 15149
I20260807 06:27:19.798352 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:27:19.798687 16376 external_mini_cluster.cc:1428] Setting key 370c39ebdd027f3803f072f7d9a0dee6
I20260807 06:27:19.802007 15238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a1f34ea29c14c00b94cc7dbf024d4cc: Bootstrap starting.
I20260807 06:27:19.804314 15238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a1f34ea29c14c00b94cc7dbf024d4cc: Neither blocks nor log segments found. Creating new log.
I20260807 06:27:19.805133 15238 log.cc:824] T 00000000000000000000000000000000 P 8a1f34ea29c14c00b94cc7dbf024d4cc: Log is configured to *not* fsync() on all Append() calls
I20260807 06:27:19.807124 15238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a1f34ea29c14c00b94cc7dbf024d4cc: No bootstrap required, opened a new log
I20260807 06:27:19.807943 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:19.800190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46715 (local address 127.15.254.62:44123)
0807 06:27:19.800719 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:19.800735 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:19.800770 (+ 35us) server_negotiation.cc:407] Connection header received
0807 06:27:19.801402 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:19.801420 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:19.801780 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:19.802127 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:19.802556 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:19.803528 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:19.804459 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:19.804987 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:19.805170 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:19.806181 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:19.806186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:19.806199 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:27:19.806609 (+ 410us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:19.806661 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:27:19.806685 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:19.806805 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:19.806977 (+ 172us) server_negotiation.cc:299] Negotiation successful
0807 06:27:19.807203 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":152,"threads_started":1}
I20260807 06:27:19.811054 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:27:19.811009 15238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a1f34ea29c14c00b94cc7dbf024d4cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a1f34ea29c14c00b94cc7dbf024d4cc" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 44123 } }
I20260807 06:27:19.811287 15238 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a1f34ea29c14c00b94cc7dbf024d4cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:27:19.811367 15238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a1f34ea29c14c00b94cc7dbf024d4cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a1f34ea29c14c00b94cc7dbf024d4cc, State: Initialized, Role: FOLLOWER
I20260807 06:27:19.811851 15238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a1f34ea29c14c00b94cc7dbf024d4cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a1f34ea29c14c00b94cc7dbf024d4cc" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 44123 } }
I20260807 06:27:19.811988 15238 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a1f34ea29c14c00b94cc7dbf024d4cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:27:19.812081 15238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a1f34ea29c14c00b94cc7dbf024d4cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:27:19.812175 15238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a1f34ea29c14c00b94cc7dbf024d4cc [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:27:19.813297 15238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a1f34ea29c14c00b94cc7dbf024d4cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a1f34ea29c14c00b94cc7dbf024d4cc" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 44123 } }
I20260807 06:27:19.813714 15238 leader_election.cc:302] T 00000000000000000000000000000000 P 8a1f34ea29c14c00b94cc7dbf024d4cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a1f34ea29c14c00b94cc7dbf024d4cc; no voters:
I20260807 06:27:19.814082 15238 leader_election.cc:288] T 00000000000000000000000000000000 P 8a1f34ea29c14c00b94cc7dbf024d4cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:27:19.814184 15250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a1f34ea29c14c00b94cc7dbf024d4cc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:27:19.814445 15250 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a1f34ea29c14c00b94cc7dbf024d4cc [term 1 LEADER]: Becoming Leader. State: Replica: 8a1f34ea29c14c00b94cc7dbf024d4cc, State: Running, Role: LEADER
I20260807 06:27:19.814808 15250 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a1f34ea29c14c00b94cc7dbf024d4cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a1f34ea29c14c00b94cc7dbf024d4cc" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 44123 } }
I20260807 06:27:19.815471 15238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a1f34ea29c14c00b94cc7dbf024d4cc [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:27:19.816006 15250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a1f34ea29c14c00b94cc7dbf024d4cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a1f34ea29c14c00b94cc7dbf024d4cc. Latest consensus state: current_term: 1 leader_uuid: "8a1f34ea29c14c00b94cc7dbf024d4cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a1f34ea29c14c00b94cc7dbf024d4cc" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 44123 } } }
I20260807 06:27:19.816130 15250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a1f34ea29c14c00b94cc7dbf024d4cc [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:19.816517 15256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:27:19.817083 15251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a1f34ea29c14c00b94cc7dbf024d4cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a1f34ea29c14c00b94cc7dbf024d4cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a1f34ea29c14c00b94cc7dbf024d4cc" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 44123 } } }
I20260807 06:27:19.817191 15251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a1f34ea29c14c00b94cc7dbf024d4cc [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:19.818845 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:19.813522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43025 (local address 127.15.254.62:44123)
0807 06:27:19.813667 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:19.813670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:19.813911 (+ 241us) server_negotiation.cc:407] Connection header received
0807 06:27:19.814064 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:19.814095 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:19.814154 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:19.814235 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:19.814661 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:19.815407 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:19.816584 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:19.816918 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:19.817041 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:19.818189 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:19.818193 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:19.818196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:27:19.818209 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:19.818238 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:27:19.818250 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:19.818315 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:19.818650 (+ 335us) server_negotiation.cc:299] Negotiation successful
0807 06:27:19.818708 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 06:27:19.820183 15256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:27:19.826287 15256 catalog_manager.cc:1347] Generated new cluster ID: 717428c803224db4a423a906ce06471b
I20260807 06:27:19.826370 15256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:27:19.895376 15256 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:27:19.896535 15256 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:27:19.905570 15256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a1f34ea29c14c00b94cc7dbf024d4cc: Generated new TSK 0
I20260807 06:27:19.906545 15256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:27:19.923823 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 15149
2026-08-07T06:27:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1722 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:27:19 dist-test-slave-44d2 krb5kdc[15273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:27:19 dist-test-slave-44d2 krb5kdc[15273](info): set up 2 sockets
Aug 07 06:27:19 dist-test-slave-44d2 krb5kdc[15273](info): commencing operation
krb5kdc: starting...
W20260807 06:27:21.151027 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.200s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:27:21 dist-test-slave-44d2 krb5kdc[15273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:27:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:27:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:27:21.294440 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:36745
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:39911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:36745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:27:21.410789 15295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:27:21.411115 15295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:27:21.411187 15295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:27:21.415210 15295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:27:21.415321 15295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:27:21.415356 15295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:27:21.415398 15295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:27:21.415434 15295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:27:21.420653 15295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:39911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:36745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:36745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:27:21.421723 15295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:27:21.422842 15295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:27:21.431527 15306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:21.431510 15302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:21.431632 15300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:27:21.431689 15295 server_base.cc:1034] running on GCE node
I20260807 06:27:21.432382 15295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:27:21.433544 15295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:27:21.434721 15295 hybrid_clock.cc:648] HybridClock initialized: now 1786084041434690 us; error 42 us; skew 500 ppm
Aug 07 06:27:21 dist-test-slave-44d2 krb5kdc[15273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084041, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:27:21.443456 15295 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:27:21.444455 15295 webserver.cc:460] Webserver started at http://127.15.254.62:40255/ using document root <none> and password file <none>
I20260807 06:27:21.445081 15295 fs_manager.cc:359] Metadata directory not provided
I20260807 06:27:21.445147 15295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:27:21.445322 15295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:27:21.447494 15295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "f873615d60334135a9f30a9f13ed5361"
format_stamp: "Formatted at 2026-08-07 06:27:21 on dist-test-slave-44d2"
server_key: "3867c6c08bcae61aa92ad00558807109"
server_key_iv: "32eeb88bc7d852ee28234374e5f08c29"
server_key_version: "encryptionkey@0"
I20260807 06:27:21.448026 15295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "f873615d60334135a9f30a9f13ed5361"
format_stamp: "Formatted at 2026-08-07 06:27:21 on dist-test-slave-44d2"
server_key: "3867c6c08bcae61aa92ad00558807109"
server_key_iv: "32eeb88bc7d852ee28234374e5f08c29"
server_key_version: "encryptionkey@0"
I20260807 06:27:21.452231 15295 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260807 06:27:21.455083 15316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:27:21.456609 15295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:27:21.456753 15295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "f873615d60334135a9f30a9f13ed5361"
format_stamp: "Formatted at 2026-08-07 06:27:21 on dist-test-slave-44d2"
server_key: "3867c6c08bcae61aa92ad00558807109"
server_key_iv: "32eeb88bc7d852ee28234374e5f08c29"
server_key_version: "encryptionkey@0"
I20260807 06:27:21.456876 15295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:27:21.461809 15295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:27:21.465680 15295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:27:21.465900 15295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:27:21.473619 15295 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:36745
I20260807 06:27:21.473641 15396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:36745 every 8 connection(s)
I20260807 06:27:21.474624 15295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:27:21.477594 15397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:27:21.482247 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 15295
I20260807 06:27:21.482363 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:27:21.482712 16376 external_mini_cluster.cc:1428] Setting key 124deceaa1e0cc308300fa2f72aa5b23
I20260807 06:27:21.483553 15397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f873615d60334135a9f30a9f13ed5361: Bootstrap starting.
I20260807 06:27:21.486088 15397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f873615d60334135a9f30a9f13ed5361: Neither blocks nor log segments found. Creating new log.
I20260807 06:27:21.487140 15397 log.cc:824] T 00000000000000000000000000000000 P f873615d60334135a9f30a9f13ed5361: Log is configured to *not* fsync() on all Append() calls
I20260807 06:27:21.488451 15401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:21.484245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54843 (local address 127.15.254.62:36745)
0807 06:27:21.484721 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:21.484737 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:21.484774 (+ 37us) server_negotiation.cc:407] Connection header received
0807 06:27:21.485484 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:21.485500 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:21.485790 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:21.486142 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:21.487053 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:21.487058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:21.487069 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:27:21.487080 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:21.487134 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:27:21.487154 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:21.487270 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:21.487378 (+ 108us) server_negotiation.cc:299] Negotiation successful
0807 06:27:21.487566 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":161,"threads_started":1}
I20260807 06:27:21.489599 15397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f873615d60334135a9f30a9f13ed5361: No bootstrap required, opened a new log
I20260807 06:27:21.492385 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:27:21.494837 15397 raft_consensus.cc:348] T 00000000000000000000000000000000 P f873615d60334135a9f30a9f13ed5361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f873615d60334135a9f30a9f13ed5361" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36745 } }
I20260807 06:27:21.495249 15397 raft_consensus.cc:374] T 00000000000000000000000000000000 P f873615d60334135a9f30a9f13ed5361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:27:21.495325 15397 raft_consensus.cc:729] T 00000000000000000000000000000000 P f873615d60334135a9f30a9f13ed5361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f873615d60334135a9f30a9f13ed5361, State: Initialized, Role: FOLLOWER
I20260807 06:27:21.495910 15397 consensus_queue.cc:260] T 00000000000000000000000000000000 P f873615d60334135a9f30a9f13ed5361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f873615d60334135a9f30a9f13ed5361" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36745 } }
I20260807 06:27:21.496101 15397 raft_consensus.cc:388] T 00000000000000000000000000000000 P f873615d60334135a9f30a9f13ed5361 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:27:21.496229 15397 raft_consensus.cc:482] T 00000000000000000000000000000000 P f873615d60334135a9f30a9f13ed5361 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:27:21.496363 15397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f873615d60334135a9f30a9f13ed5361 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:27:21.497521 15401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:21.495097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52275 (local address 127.15.254.62:36745)
0807 06:27:21.495212 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:21.495217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:21.495499 (+ 282us) server_negotiation.cc:407] Connection header received
0807 06:27:21.495801 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:21.495804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:21.495862 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:21.495948 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:21.497057 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:21.497081 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:21.497086 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:27:21.497099 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:21.497128 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:27:21.497140 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:21.497230 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:21.497356 (+ 126us) server_negotiation.cc:299] Negotiation successful
0807 06:27:21.497398 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 06:27:21.499949 15397 raft_consensus.cc:504] T 00000000000000000000000000000000 P f873615d60334135a9f30a9f13ed5361 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f873615d60334135a9f30a9f13ed5361" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36745 } }
I20260807 06:27:21.500366 15397 leader_election.cc:302] T 00000000000000000000000000000000 P f873615d60334135a9f30a9f13ed5361 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f873615d60334135a9f30a9f13ed5361; no voters:
I20260807 06:27:21.500836 15397 leader_election.cc:288] T 00000000000000000000000000000000 P f873615d60334135a9f30a9f13ed5361 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:27:21.500964 15411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f873615d60334135a9f30a9f13ed5361 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:27:21.501266 15411 raft_consensus.cc:686] T 00000000000000000000000000000000 P f873615d60334135a9f30a9f13ed5361 [term 1 LEADER]: Becoming Leader. State: Replica: f873615d60334135a9f30a9f13ed5361, State: Running, Role: LEADER
I20260807 06:27:21.501616 15411 consensus_queue.cc:237] T 00000000000000000000000000000000 P f873615d60334135a9f30a9f13ed5361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f873615d60334135a9f30a9f13ed5361" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36745 } }
I20260807 06:27:21.502291 15397 sys_catalog.cc:564] T 00000000000000000000000000000000 P f873615d60334135a9f30a9f13ed5361 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:27:21.502861 15411 sys_catalog.cc:455] T 00000000000000000000000000000000 P f873615d60334135a9f30a9f13ed5361 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f873615d60334135a9f30a9f13ed5361. Latest consensus state: current_term: 1 leader_uuid: "f873615d60334135a9f30a9f13ed5361" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f873615d60334135a9f30a9f13ed5361" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36745 } } }
I20260807 06:27:21.502983 15411 sys_catalog.cc:458] T 00000000000000000000000000000000 P f873615d60334135a9f30a9f13ed5361 [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:21.503376 15423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:27:21.503562 15412 sys_catalog.cc:455] T 00000000000000000000000000000000 P f873615d60334135a9f30a9f13ed5361 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f873615d60334135a9f30a9f13ed5361" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f873615d60334135a9f30a9f13ed5361" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 36745 } } }
I20260807 06:27:21.503808 15412 sys_catalog.cc:458] T 00000000000000000000000000000000 P f873615d60334135a9f30a9f13ed5361 [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:21.507036 15423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:27:21.515960 15423 catalog_manager.cc:1347] Generated new cluster ID: b968d3c8c564481387ca3c58cf56479c
I20260807 06:27:21.516073 15423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:27:21.537968 15423 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:27:21.538393 15423 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:27:21.550800 15423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f873615d60334135a9f30a9f13ed5361: Generated new TSK 0
I20260807 06:27:21.551693 15423 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:27:21.563453 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 15295
2026-08-07T06:27:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1638 ms)
[----------] 18 tests from AuthTokenIssuingTest (29819 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:27:21 dist-test-slave-44d2 krb5kdc[15449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:27:21 dist-test-slave-44d2 krb5kdc[15449](info): set up 2 sockets
Aug 07 06:27:21 dist-test-slave-44d2 krb5kdc[15449](info): commencing operation
krb5kdc: starting...
W20260807 06:27:22.787906 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:27:22 dist-test-slave-44d2 krb5kdc[15449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:27:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:27:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:27:22.932402 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:41285
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:39367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:41285,127.15.254.61:38597,127.15.254.60:33739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:27:23.049782 15467 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 06:27:23.050117 15467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:27:23.050181 15467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:27:23.050257 15467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:27:23.054286 15467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:27:23.054368 15467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:27:23.054404 15467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:27:23.054473 15467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:27:23.054509 15467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:27:23.059782 15467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:39367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:41285,127.15.254.61:38597,127.15.254.60:33739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:41285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:27:23.060927 15467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:27:23.061968 15467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:27:23.071055 15474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:23.071038 15478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:23.071041 15476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:27:23.071357 15467 server_base.cc:1034] running on GCE node
I20260807 06:27:23.072050 15467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:27:23.073290 15467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:27:23.074501 15467 hybrid_clock.cc:648] HybridClock initialized: now 1786084043074480 us; error 42 us; skew 500 ppm
Aug 07 06:27:23 dist-test-slave-44d2 krb5kdc[15449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084043, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:27:23.077570 15467 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:27:23.078465 15467 webserver.cc:460] Webserver started at http://127.15.254.62:46555/ using document root <none> and password file <none>
I20260807 06:27:23.079082 15467 fs_manager.cc:359] Metadata directory not provided
I20260807 06:27:23.079183 15467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:27:23.079429 15467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:27:23.081570 15467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "ccaf3d026ea5494bb6bff733830f8360"
format_stamp: "Formatted at 2026-08-07 06:27:23 on dist-test-slave-44d2"
server_key: "c8e61f63cd8895a0cd9bfcaaa4864e91"
server_key_iv: "98e4038c437f24b0fab5029d8481a81f"
server_key_version: "encryptionkey@0"
I20260807 06:27:23.082161 15467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "ccaf3d026ea5494bb6bff733830f8360"
format_stamp: "Formatted at 2026-08-07 06:27:23 on dist-test-slave-44d2"
server_key: "c8e61f63cd8895a0cd9bfcaaa4864e91"
server_key_iv: "98e4038c437f24b0fab5029d8481a81f"
server_key_version: "encryptionkey@0"
I20260807 06:27:23.085887 15467 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:27:23.088439 15491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:27:23.089915 15467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:27:23.090152 15467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "ccaf3d026ea5494bb6bff733830f8360"
format_stamp: "Formatted at 2026-08-07 06:27:23 on dist-test-slave-44d2"
server_key: "c8e61f63cd8895a0cd9bfcaaa4864e91"
server_key_iv: "98e4038c437f24b0fab5029d8481a81f"
server_key_version: "encryptionkey@0"
I20260807 06:27:23.090250 15467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:27:23.113162 15467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:27:23.116938 15467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:27:23.117174 15467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:27:23.125192 15467 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:41285
I20260807 06:27:23.125504 15565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:41285 every 8 connection(s)
I20260807 06:27:23.126276 15467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:27:23.129423 15566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:27:23.130632 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 15467
I20260807 06:27:23.130803 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:27:23.131184 16376 external_mini_cluster.cc:1428] Setting key e2cc3549e7a2bf8ae7b1d6808eac64bb
I20260807 06:27:23.134836 15566 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41285 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.254.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.61@KRBTEST.COM" created.
Aug 07 06:27:23 dist-test-slave-44d2 krb5kdc[15449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084043, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:27:23.153584 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.137413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41285 (local address 127.0.0.1:57096)
0807 06:27:23.138099 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.138170 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0807 06:27:23.139526 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:27:23.140425 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:27:23.140432 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:27:23.141020 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:27:23.141468 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:23.141493 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.142878 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.142881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:23.143738 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:23.143748 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.144620 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.144623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:23.144783 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:23.145596 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:27:23.145615 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:27:23.147553 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:27:23.150164 (+ 2611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:23.150178 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:23.150190 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:23.150472 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:23.150954 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:23.150957 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:23.150959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:23.151089 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:23.151969 (+ 880us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:27:23.151975 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:27:23.152111 (+ 136us) client_negotiation.cc:769] Sending connection context
0807 06:27:23.152271 (+ 160us) client_negotiation.cc:240] Negotiation successful
0807 06:27:23.152653 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":174,"threads_started":1}
I20260807 06:27:23.153589 15572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.137413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57096 (local address 127.15.254.62:41285)
0807 06:27:23.138046 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:23.138066 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:23.138309 (+ 243us) server_negotiation.cc:407] Connection header received
0807 06:27:23.139966 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:23.139971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:23.140173 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:23.140343 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:23.141682 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.142716 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.143920 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.144418 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.144639 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:23.147721 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:23.147731 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:23.147748 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:23.147783 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:23.149993 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.150602 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.150611 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.150614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.150769 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.151204 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.151210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.151214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.151803 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:23.151980 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:23.152395 (+ 415us) server_negotiation.cc:299] Negotiation successful
0807 06:27:23.152795 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":499,"spinlock_wait_cycles":48000,"thread_start_us":174,"threads_started":1}
I20260807 06:27:23.156378 15566 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.61" port: 38597 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:27:23.157437 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.157052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:38597 (local address 127.0.0.1:60532)
0807 06:27:23.157189 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.157302 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.61:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 06:27:23.157686 15501 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.254.61:38597: connect: Connection refused (error 111)
I20260807 06:27:23.158782 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.158499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:38597 (local address 127.0.0.1:60544)
0807 06:27:23.158638 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.158686 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.61:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260807 06:27:23.159031 15566 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.61:38597: Network error: Client connection negotiation failed: client connection to 127.15.254.61:38597: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.61.keytab.
Entry for principal kudu/127.15.254.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.61.keytab.
I20260807 06:27:23.163695 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.61:38597
--webserver_interface=127.15.254.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.61.keytab
--principal=kudu/127.15.254.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:39367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:41285,127.15.254.61:38597,127.15.254.60:33739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:27:23.208315 15566 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.61" port: 38597 } attempt: 1
I20260807 06:27:23.209477 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.209078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:38597 (local address 127.0.0.1:60560)
0807 06:27:23.209272 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.209360 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.61:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260807 06:27:23.210508 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.210179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:38597 (local address 127.0.0.1:60570)
0807 06:27:23.210338 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.210421 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.61:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260807 06:27:23.210692 15566 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.61:38597: Network error: Client connection negotiation failed: client connection to 127.15.254.61:38597: connect: Connection refused (error 111)
I20260807 06:27:23.278934 15566 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.61" port: 38597 } attempt: 2
I20260807 06:27:23.279870 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.279512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:38597 (local address 127.0.0.1:60572)
0807 06:27:23.279692 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.279768 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.61:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260807 06:27:23.280814 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.280566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:38597 (local address 127.0.0.1:60580)
0807 06:27:23.280691 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.280747 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.61:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260807 06:27:23.281044 15566 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.61:38597: Network error: Client connection negotiation failed: client connection to 127.15.254.61:38597: connect: Connection refused (error 111)
W20260807 06:27:23.284029 15578 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 06:27:23.284276 15578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:27:23.284346 15578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:27:23.284415 15578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:27:23.288445 15578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:27:23.288545 15578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:27:23.288579 15578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:27:23.288614 15578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:27:23.288637 15578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:27:23.294263 15578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:39367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:41285,127.15.254.61:38597,127.15.254.60:33739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.61.keytab
--principal=kudu/127.15.254.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.61:38597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.254.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:27:23.295361 15578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:27:23.296394 15578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:27:23.305927 15585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:27:23.306090 15578 server_base.cc:1034] running on GCE node
W20260807 06:27:23.306022 15587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:23.306244 15584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:27:23.306799 15578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:27:23.307946 15578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:27:23.309165 15578 hybrid_clock.cc:648] HybridClock initialized: now 1786084043309144 us; error 32 us; skew 500 ppm
Aug 07 06:27:23 dist-test-slave-44d2 krb5kdc[15449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084043, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:27:23.312254 15578 init.cc:377] Logged in from keytab as kudu/127.15.254.61@KRBTEST.COM (short username kudu)
I20260807 06:27:23.313290 15578 webserver.cc:460] Webserver started at http://127.15.254.61:38929/ using document root <none> and password file <none>
I20260807 06:27:23.313898 15578 fs_manager.cc:359] Metadata directory not provided
I20260807 06:27:23.313985 15578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:27:23.314282 15578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:27:23.316465 15578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-1/data/instance:
uuid: "f529a84bf69d4bbeb3377bf898056181"
format_stamp: "Formatted at 2026-08-07 06:27:23 on dist-test-slave-44d2"
server_key: "afdb12a99d4811597e6c4dff1653c022"
server_key_iv: "7538adfc36cfca4872cdea12b1ed199e"
server_key_version: "encryptionkey@0"
I20260807 06:27:23.317152 15578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-1/wal/instance:
uuid: "f529a84bf69d4bbeb3377bf898056181"
format_stamp: "Formatted at 2026-08-07 06:27:23 on dist-test-slave-44d2"
server_key: "afdb12a99d4811597e6c4dff1653c022"
server_key_iv: "7538adfc36cfca4872cdea12b1ed199e"
server_key_version: "encryptionkey@0"
I20260807 06:27:23.321059 15578 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260807 06:27:23.323709 15596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:27:23.325126 15578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:27:23.325263 15578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-1/wal
uuid: "f529a84bf69d4bbeb3377bf898056181"
format_stamp: "Formatted at 2026-08-07 06:27:23 on dist-test-slave-44d2"
server_key: "afdb12a99d4811597e6c4dff1653c022"
server_key_iv: "7538adfc36cfca4872cdea12b1ed199e"
server_key_version: "encryptionkey@0"
I20260807 06:27:23.325376 15578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:27:23.342345 15578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:27:23.345638 15578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:27:23.345865 15578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:27:23.353591 15578 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.61:38597
I20260807 06:27:23.353955 15668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.61:38597 every 8 connection(s)
I20260807 06:27:23.354629 15578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-1/data/info.pb
I20260807 06:27:23.357602 15671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:27:23.361590 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 15578
I20260807 06:27:23.361737 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-1/wal/instance
I20260807 06:27:23.361472 15671 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41285 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:27:23.362046 16376 external_mini_cluster.cc:1428] Setting key 85f13883b7623b73544667d53c79ea08
WARNING: no policy specified for kudu/127.15.254.60@KRBTEST.COM; defaulting to no policy
I20260807 06:27:23.372256 15566 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.61" port: 38597 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.60@KRBTEST.COM" created.
Aug 07 06:27:23 dist-test-slave-44d2 krb5kdc[15449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084043, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.61@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
Aug 07 06:27:23 dist-test-slave-44d2 krb5kdc[15449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084043, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for kudu/127.15.254.61@KRBTEST.COM
I20260807 06:27:23.380620 15572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.363636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57100 (local address 127.15.254.62:41285)
0807 06:27:23.363780 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:23.363783 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:23.364888 (+ 1105us) server_negotiation.cc:407] Connection header received
0807 06:27:23.365844 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:23.365921 (+ 77us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:23.366003 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:23.366116 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:23.367425 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.368109 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.369374 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.369614 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.369727 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:23.375918 (+ 6191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:23.375921 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:23.375924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:23.375948 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:23.378090 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.378839 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.378843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.378845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.378930 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.379301 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.379304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.379306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.379496 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:23.379606 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:23.380274 (+ 668us) server_negotiation.cc:299] Negotiation successful
0807 06:27:23.380412 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 06:27:23.381665 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.364112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41285 (local address 127.0.0.1:57100)
0807 06:27:23.364705 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.364760 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 06:27:23.365624 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:27:23.366276 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:27:23.366283 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:27:23.366674 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:27:23.367259 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:23.367276 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.368252 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.368256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:23.369243 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:23.369254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.369773 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.369777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:23.369953 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:23.370641 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:27:23.370659 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:27:23.375755 (+ 5096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:27:23.378242 (+ 2487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:23.378252 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:23.378274 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:23.378717 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:23.379029 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:23.379033 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:23.379035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:23.379191 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:23.379607 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:27:23.379615 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:27:23.379968 (+ 353us) client_negotiation.cc:769] Sending connection context
0807 06:27:23.380202 (+ 234us) client_negotiation.cc:240] Negotiation successful
0807 06:27:23.380581 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":171,"threads_started":1}
I20260807 06:27:23.383270 15671 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.61" port: 38597 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:27:23.384632 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.372814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:38597 (local address 127.0.0.1:60582)
0807 06:27:23.372941 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.372957 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:27:23.373087 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:27:23.373691 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:27:23.373693 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:27:23.373844 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:27:23.373945 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:23.373950 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.375064 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.375068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:23.375883 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:23.375899 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.376716 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.376723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:23.376870 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:23.377499 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:27:23.377519 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:27:23.379883 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:27:23.382793 (+ 2910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:23.382796 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:23.382798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:23.383071 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:23.383364 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:23.383367 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:23.383369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:23.383458 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:23.384047 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:27:23.384052 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:27:23.384157 (+ 105us) client_negotiation.cc:769] Sending connection context
0807 06:27:23.384266 (+ 109us) client_negotiation.cc:240] Negotiation successful
0807 06:27:23.384418 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260807 06:27:23.385548 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.372977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60582 (local address 127.15.254.61:38597)
0807 06:27:23.373296 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:23.373310 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:23.373332 (+ 22us) server_negotiation.cc:407] Connection header received
0807 06:27:23.373370 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:23.373374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:23.373539 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:23.373688 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:23.374090 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.374940 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.376020 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.376579 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.376749 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:23.380046 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:23.380060 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:23.380079 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:23.380114 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:23.382675 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.383171 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.383176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.383178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.383266 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.383553 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.383557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.383559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.383870 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:23.383999 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:23.384671 (+ 672us) server_negotiation.cc:299] Negotiation successful
0807 06:27:23.384840 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":112,"threads_started":1}
I20260807 06:27:23.387329 15566 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:27:23.388073 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.387809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:33739 (local address 127.0.0.1:54926)
0807 06:27:23.387932 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.387979 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:33739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260807 06:27:23.388957 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.388629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:33739 (local address 127.0.0.1:54932)
0807 06:27:23.388818 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.388886 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:33739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260807 06:27:23.389201 15566 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.60:33739: Network error: Client connection negotiation failed: client connection to 127.15.254.60:33739: connect: Connection refused (error 111)
Aug 07 06:27:23 dist-test-slave-44d2 krb5kdc[15449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084043, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.61@KRBTEST.COM for kudu/127.15.254.61@KRBTEST.COM
I20260807 06:27:23.395002 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.383862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:38597 (local address 127.0.0.1:60596)
0807 06:27:23.384045 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.384059 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:27:23.384125 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:27:23.384600 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:27:23.384603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:27:23.385080 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:27:23.385217 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:23.385227 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.386883 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.386886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:23.387523 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:23.387534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.388166 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.388171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:23.388360 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:23.388893 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:27:23.388908 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:27:23.390537 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:27:23.393278 (+ 2741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:23.393285 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:23.393288 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:23.393601 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:23.393942 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:23.393945 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:23.393946 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:23.394033 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:23.394473 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:27:23.394479 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:27:23.394584 (+ 105us) client_negotiation.cc:769] Sending connection context
0807 06:27:23.394679 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 06:27:23.394831 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":43}
I20260807 06:27:23.395134 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.383960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60596 (local address 127.15.254.61:38597)
0807 06:27:23.384345 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:23.384370 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:23.384384 (+ 14us) server_negotiation.cc:407] Connection header received
0807 06:27:23.384417 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:23.384419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:23.384459 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:23.384803 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:23.385349 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.386711 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.387652 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.388018 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.388154 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:23.390793 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:23.390797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:23.390799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:23.390842 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:23.393117 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.393745 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.393748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.393749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.393866 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.394122 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.394124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.394125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.394333 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:23.394438 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:23.394732 (+ 294us) server_negotiation.cc:299] Negotiation successful
0807 06:27:23.394883 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":327,"thread_start_us":134,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.60.keytab.
Entry for principal kudu/127.15.254.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.60.keytab.
I20260807 06:27:23.396193 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.60:33739
--webserver_interface=127.15.254.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.60.keytab
--principal=kudu/127.15.254.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:39367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:41285,127.15.254.61:38597,127.15.254.60:33739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:27:23.396246 15671 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:27:23.396981 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.396735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:33739 (local address 127.0.0.1:54936)
0807 06:27:23.396852 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.396927 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:33739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 06:27:23.397192 15600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.254.60:33739: connect: Connection refused (error 111)
I20260807 06:27:23.398139 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.397890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:33739 (local address 127.0.0.1:54940)
0807 06:27:23.397992 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.398059 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:33739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260807 06:27:23.398336 15671 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.60:33739: Network error: Client connection negotiation failed: client connection to 127.15.254.60:33739: connect: Connection refused (error 111)
I20260807 06:27:23.420413 15566 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } attempt: 1
I20260807 06:27:23.421444 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.421121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:33739 (local address 127.0.0.1:54942)
0807 06:27:23.421287 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.421362 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:33739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260807 06:27:23.422298 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.422072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:33739 (local address 127.0.0.1:54948)
0807 06:27:23.422189 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.422242 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:33739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 06:27:23.422433 15566 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.60:33739: Network error: Client connection negotiation failed: client connection to 127.15.254.60:33739: connect: Connection refused (error 111)
I20260807 06:27:23.447642 15671 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } attempt: 1
I20260807 06:27:23.448724 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.448326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:33739 (local address 127.0.0.1:54950)
0807 06:27:23.448564 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.448635 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:33739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260807 06:27:23.449524 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.449290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:33739 (local address 127.0.0.1:54966)
0807 06:27:23.449411 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.449466 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:33739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 06:27:23.449673 15671 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.60:33739: Network error: Client connection negotiation failed: client connection to 127.15.254.60:33739: connect: Connection refused (error 111)
I20260807 06:27:23.497596 15566 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } attempt: 2
I20260807 06:27:23.498632 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.498206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:33739 (local address 127.0.0.1:54982)
0807 06:27:23.498475 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.498544 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:33739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260807 06:27:23.499464 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.499217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:33739 (local address 127.0.0.1:54988)
0807 06:27:23.499334 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.499399 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:33739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260807 06:27:23.499606 15566 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.60:33739: Network error: Client connection negotiation failed: client connection to 127.15.254.60:33739: connect: Connection refused (error 111)
I20260807 06:27:23.517845 15671 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } attempt: 2
I20260807 06:27:23.518687 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.518386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:33739 (local address 127.0.0.1:54996)
0807 06:27:23.518536 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.518607 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:33739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 06:27:23.518594 15683 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 06:27:23.518970 15683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:27:23.519032 15683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:27:23.519089 15683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260807 06:27:23.519506 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.519283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:33739 (local address 127.0.0.1:55002)
0807 06:27:23.519391 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.519445 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:33739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260807 06:27:23.519644 15671 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.60:33739: Network error: Client connection negotiation failed: client connection to 127.15.254.60:33739: connect: Connection refused (error 111)
W20260807 06:27:23.523325 15683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:27:23.523409 15683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:27:23.523458 15683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:27:23.523504 15683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:27:23.523541 15683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:27:23.529009 15683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:39367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:41285,127.15.254.61:38597,127.15.254.60:33739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.60.keytab
--principal=kudu/127.15.254.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.60:33739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.254.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:27:23.530247 15683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:27:23.531358 15683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:27:23.540845 15692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:23.540845 15688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:23.540850 15689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:27:23.541100 15683 server_base.cc:1034] running on GCE node
I20260807 06:27:23.541648 15683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:27:23.542809 15683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:27:23.543977 15683 hybrid_clock.cc:648] HybridClock initialized: now 1786084043543958 us; error 35 us; skew 500 ppm
Aug 07 06:27:23 dist-test-slave-44d2 krb5kdc[15449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084043, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:27:23.546916 15683 init.cc:377] Logged in from keytab as kudu/127.15.254.60@KRBTEST.COM (short username kudu)
I20260807 06:27:23.547858 15683 webserver.cc:460] Webserver started at http://127.15.254.60:39133/ using document root <none> and password file <none>
I20260807 06:27:23.548406 15683 fs_manager.cc:359] Metadata directory not provided
I20260807 06:27:23.548467 15683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:27:23.548731 15683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:27:23.550827 15683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-2/data/instance:
uuid: "805c291c3cdf48caac286f27afdd785c"
format_stamp: "Formatted at 2026-08-07 06:27:23 on dist-test-slave-44d2"
server_key: "e1a632ee8b504132c780274b8d07f8c4"
server_key_iv: "b8e8ac7348c019f006a8bc362f78eb6b"
server_key_version: "encryptionkey@0"
I20260807 06:27:23.551342 15683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-2/wal/instance:
uuid: "805c291c3cdf48caac286f27afdd785c"
format_stamp: "Formatted at 2026-08-07 06:27:23 on dist-test-slave-44d2"
server_key: "e1a632ee8b504132c780274b8d07f8c4"
server_key_iv: "b8e8ac7348c019f006a8bc362f78eb6b"
server_key_version: "encryptionkey@0"
I20260807 06:27:23.555147 15683 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:27:23.557763 15700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:27:23.559101 15683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 06:27:23.559211 15683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-2/wal
uuid: "805c291c3cdf48caac286f27afdd785c"
format_stamp: "Formatted at 2026-08-07 06:27:23 on dist-test-slave-44d2"
server_key: "e1a632ee8b504132c780274b8d07f8c4"
server_key_iv: "b8e8ac7348c019f006a8bc362f78eb6b"
server_key_version: "encryptionkey@0"
I20260807 06:27:23.559299 15683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:27:23.581175 15683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:27:23.584542 15683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:27:23.584712 15683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:27:23.592579 15683 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.60:33739
I20260807 06:27:23.593003 15770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.60:33739 every 8 connection(s)
I20260807 06:27:23.593725 15683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-2/data/info.pb
I20260807 06:27:23.594785 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 15683
I20260807 06:27:23.594925 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786083738959718-16376-0/minicluster-data/master-2/wal/instance
I20260807 06:27:23.595249 16376 external_mini_cluster.cc:1428] Setting key cb8c18c4a17a6b18edaa0d61a72dd2ee
I20260807 06:27:23.597469 15771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:27:23.598796 15566 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } attempt: 3
I20260807 06:27:23.602468 15771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41285 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:27:23 dist-test-slave-44d2 krb5kdc[15449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084043, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for kudu/127.15.254.60@KRBTEST.COM
Aug 07 06:27:23 dist-test-slave-44d2 krb5kdc[15449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.60@KRBTEST.COM
I20260807 06:27:23.610790 15671 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } attempt: 3
Aug 07 06:27:23 dist-test-slave-44d2 krb5kdc[15449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084043, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.60@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:27:23.611279 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.599306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:33739 (local address 127.0.0.1:55016)
0807 06:27:23.599450 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.599471 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 06:27:23.599561 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:27:23.600009 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:27:23.600012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:27:23.600220 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:27:23.600327 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:23.600333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.601614 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.601621 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:23.602485 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:23.602498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.603501 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.603505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:23.603591 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:23.604125 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:27:23.604140 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:27:23.605798 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:27:23.608998 (+ 3200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:23.609002 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:23.609004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:23.609278 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:23.609993 (+ 715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:23.609999 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:23.610001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:23.610105 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:23.610687 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:27:23.610690 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:27:23.610820 (+ 130us) client_negotiation.cc:769] Sending connection context
0807 06:27:23.610934 (+ 114us) client_negotiation.cc:240] Negotiation successful
0807 06:27:23.611079 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 06:27:23.612850 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.596717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55008 (local address 127.15.254.60:33739)
0807 06:27:23.597254 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:23.597280 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:23.597330 (+ 50us) server_negotiation.cc:407] Connection header received
0807 06:27:23.598120 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:23.598140 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:23.598502 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:23.598872 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:23.600274 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.601547 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.602748 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.603344 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.603532 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:23.606624 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:23.606627 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:23.606629 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:23.606655 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:23.608881 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.609419 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.609426 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.609434 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.609614 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.610039 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.610043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.610044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.610570 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:23.610787 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:23.611339 (+ 552us) server_negotiation.cc:299] Negotiation successful
0807 06:27:23.611666 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":220,"server-negotiator.queue_time_us":349,"thread_start_us":150,"threads_started":1}
I20260807 06:27:23.613276 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.599353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55016 (local address 127.15.254.60:33739)
0807 06:27:23.599710 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:23.599719 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:23.599735 (+ 16us) server_negotiation.cc:407] Connection header received
0807 06:27:23.599771 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:23.599774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:23.599833 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:23.599947 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:23.600467 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.601479 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.602625 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.603349 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.603851 (+ 502us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:23.605982 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:23.605990 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:23.606005 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:23.606036 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:23.608833 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.609729 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.609732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.609734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.609805 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.610206 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.610211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.610213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.610566 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:23.610715 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:23.612283 (+ 1568us) server_negotiation.cc:299] Negotiation successful
0807 06:27:23.612755 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":291,"thread_start_us":140,"threads_started":1}
I20260807 06:27:23.616070 15572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.604698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57102 (local address 127.15.254.62:41285)
0807 06:27:23.604844 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:23.604851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:23.605288 (+ 437us) server_negotiation.cc:407] Connection header received
0807 06:27:23.605527 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:23.605531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:23.605571 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:23.605657 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:23.606535 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.607285 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.608498 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.608882 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.608993 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:23.612092 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:23.612096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:23.612098 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:23.612123 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:23.613865 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.614550 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.614554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.614555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.614650 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.614992 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.614996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.614998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.615225 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:23.615304 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:23.615796 (+ 492us) server_negotiation.cc:299] Negotiation successful
0807 06:27:23.615918 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64,"spinlock_wait_cycles":3968}
I20260807 06:27:23.616703 15778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.604612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:41285 (local address 127.0.0.1:57102)
0807 06:27:23.605155 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.605210 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 06:27:23.605395 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:27:23.605713 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:27:23.605721 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:27:23.606247 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:27:23.606412 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:23.606420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.607423 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.607427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:23.608329 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:23.608344 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.609121 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.609124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:23.609215 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:23.609636 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:27:23.609656 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:27:23.611936 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:27:23.613988 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:23.613996 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:23.614000 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:23.614273 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:23.614772 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:23.614775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:23.614776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:23.614884 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:23.615402 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:27:23.615411 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:27:23.615547 (+ 136us) client_negotiation.cc:769] Sending connection context
0807 06:27:23.616321 (+ 774us) client_negotiation.cc:240] Negotiation successful
0807 06:27:23.616544 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":117,"threads_started":1}
I20260807 06:27:23.618541 15566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360: Bootstrap starting.
I20260807 06:27:23.621065 15566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360: Neither blocks nor log segments found. Creating new log.
I20260807 06:27:23.621193 15771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.61" port: 38597 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:27:23 dist-test-slave-44d2 krb5kdc[15449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
Aug 07 06:27:23 dist-test-slave-44d2 krb5kdc[15449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084043, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.61@KRBTEST.COM for kudu/127.15.254.60@KRBTEST.COM
I20260807 06:27:23.622624 15566 log.cc:824] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360: Log is configured to *not* fsync() on all Append() calls
I20260807 06:27:23.626099 15566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360: No bootstrap required, opened a new log
I20260807 06:27:23.626632 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.611277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:33739 (local address 127.0.0.1:55026)
0807 06:27:23.611395 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.611412 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:27:23.611489 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:27:23.612316 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:27:23.612319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:27:23.612599 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:27:23.612737 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:23.612745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.613842 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.613847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:23.614404 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:23.614438 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.615113 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.615117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:23.615217 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:23.620750 (+ 5533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:27:23.620769 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:27:23.622372 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:27:23.624786 (+ 2414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:23.624790 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:23.624792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:23.625180 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:23.625495 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:23.625498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:23.625502 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:23.625598 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:23.626098 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:27:23.626102 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:27:23.626200 (+ 98us) client_negotiation.cc:769] Sending connection context
0807 06:27:23.626311 (+ 111us) client_negotiation.cc:240] Negotiation successful
0807 06:27:23.626455 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260807 06:27:23.626896 15572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.616627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57112 (local address 127.15.254.62:41285)
0807 06:27:23.616740 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:23.616744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:23.616909 (+ 165us) server_negotiation.cc:407] Connection header received
0807 06:27:23.616987 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:23.616990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:23.617038 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:23.617124 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:23.617757 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.618247 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.619075 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.619295 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.619401 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:23.622592 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:23.622596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:23.622600 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:23.622628 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:23.624529 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.625050 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.625054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.625055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.625138 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.625490 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.625493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.625495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.625735 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:23.626551 (+ 816us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:23.626626 (+ 75us) server_negotiation.cc:299] Negotiation successful
0807 06:27:23.626730 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Aug 07 06:27:23 dist-test-slave-44d2 krb5kdc[15449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084043, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.60@KRBTEST.COM for kudu/127.15.254.61@KRBTEST.COM
I20260807 06:27:23.627454 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.611738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55026 (local address 127.15.254.60:33739)
0807 06:27:23.612029 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:23.612039 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:23.612052 (+ 13us) server_negotiation.cc:407] Connection header received
0807 06:27:23.612086 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:23.612091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:23.612141 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:23.612853 (+ 712us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:23.612965 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.613730 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.614601 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.614958 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.615071 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:23.622539 (+ 7468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:23.622545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:23.622552 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:23.622580 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:23.624672 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.625295 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.625298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.625300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.625381 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.625741 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.625744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.625746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.625975 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:23.627059 (+ 1084us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:23.627135 (+ 76us) server_negotiation.cc:299] Negotiation successful
0807 06:27:23.627271 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":95,"threads_started":1}
I20260807 06:27:23.630823 15671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f529a84bf69d4bbeb3377bf898056181: Bootstrap starting.
I20260807 06:27:23.631850 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.621835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60604 (local address 127.15.254.61:38597)
0807 06:27:23.621962 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:23.621967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:23.621982 (+ 15us) server_negotiation.cc:407] Connection header received
0807 06:27:23.622029 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:23.622033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:23.622078 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:23.622161 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:23.622645 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.623398 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.624610 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.625002 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.625086 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:23.627796 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:23.627800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:23.627803 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:23.627827 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:23.629920 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.630412 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.630427 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.630429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.630515 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.630839 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.630845 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.630848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.631134 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:23.631211 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:23.631593 (+ 382us) server_negotiation.cc:299] Negotiation successful
0807 06:27:23.631710 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 06:27:23.632087 15566 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaf3d026ea5494bb6bff733830f8360" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41285 } } peers { permanent_uuid: "f529a84bf69d4bbeb3377bf898056181" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 38597 } } peers { permanent_uuid: "805c291c3cdf48caac286f27afdd785c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } }
I20260807 06:27:23.632306 15566 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccaf3d026ea5494bb6bff733830f8360, State: Initialized, Role: FOLLOWER
I20260807 06:27:23.632361 15778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.621751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:38597 (local address 127.0.0.1:60604)
0807 06:27:23.621863 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.621879 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:27:23.621960 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:27:23.622192 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:27:23.622195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:27:23.622368 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:27:23.622503 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:23.622511 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.623576 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.623581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:23.624388 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:23.624401 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.625265 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.625268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:23.625386 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:23.625816 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:27:23.625842 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:27:23.627498 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:27:23.630039 (+ 2541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:23.630042 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:23.630044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:23.630290 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:23.630638 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:23.630640 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:23.630642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:23.630729 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:23.631297 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:27:23.631300 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:27:23.631392 (+ 92us) client_negotiation.cc:769] Sending connection context
0807 06:27:23.632020 (+ 628us) client_negotiation.cc:240] Negotiation successful
0807 06:27:23.632181 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"spinlock_wait_cycles":8576}
I20260807 06:27:23.632701 15566 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaf3d026ea5494bb6bff733830f8360" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41285 } } peers { permanent_uuid: "f529a84bf69d4bbeb3377bf898056181" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 38597 } } peers { permanent_uuid: "805c291c3cdf48caac286f27afdd785c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } }
I20260807 06:27:23.633034 15771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:27:23.633812 15671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f529a84bf69d4bbeb3377bf898056181: Neither blocks nor log segments found. Creating new log.
I20260807 06:27:23.634294 15566 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:27:23.634647 15671 log.cc:824] T 00000000000000000000000000000000 P f529a84bf69d4bbeb3377bf898056181: Log is configured to *not* fsync() on all Append() calls
Aug 07 06:27:23 dist-test-slave-44d2 krb5kdc[15449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.61@KRBTEST.COM
I20260807 06:27:23.636865 15671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f529a84bf69d4bbeb3377bf898056181: No bootstrap required, opened a new log
W20260807 06:27:23.640297 15799 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:27:23.640594 15799 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:27:23.640645 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.628100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60608 (local address 127.15.254.61:38597)
0807 06:27:23.628221 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:23.628224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:23.628970 (+ 746us) server_negotiation.cc:407] Connection header received
0807 06:27:23.629017 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:23.629020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:23.629062 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:23.629126 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:23.630781 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.631614 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.632679 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.633061 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.633176 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:23.636294 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:23.636300 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:23.636306 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:23.636336 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:23.638224 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.638743 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.638746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.638748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.638860 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.639181 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.639184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.639186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.639400 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:23.639588 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:23.640320 (+ 732us) server_negotiation.cc:299] Negotiation successful
0807 06:27:23.640454 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 06:27:23.641192 15671 raft_consensus.cc:348] T 00000000000000000000000000000000 P f529a84bf69d4bbeb3377bf898056181 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaf3d026ea5494bb6bff733830f8360" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41285 } } peers { permanent_uuid: "f529a84bf69d4bbeb3377bf898056181" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 38597 } } peers { permanent_uuid: "805c291c3cdf48caac286f27afdd785c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } }
I20260807 06:27:23.641516 15671 raft_consensus.cc:729] T 00000000000000000000000000000000 P f529a84bf69d4bbeb3377bf898056181 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f529a84bf69d4bbeb3377bf898056181, State: Initialized, Role: FOLLOWER
I20260807 06:27:23.641902 15671 consensus_queue.cc:260] T 00000000000000000000000000000000 P f529a84bf69d4bbeb3377bf898056181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaf3d026ea5494bb6bff733830f8360" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41285 } } peers { permanent_uuid: "f529a84bf69d4bbeb3377bf898056181" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 38597 } } peers { permanent_uuid: "805c291c3cdf48caac286f27afdd785c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } }
I20260807 06:27:23.642486 15804 sys_catalog.cc:455] T 00000000000000000000000000000000 P f529a84bf69d4bbeb3377bf898056181 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaf3d026ea5494bb6bff733830f8360" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41285 } } peers { permanent_uuid: "f529a84bf69d4bbeb3377bf898056181" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 38597 } } peers { permanent_uuid: "805c291c3cdf48caac286f27afdd785c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } } }
I20260807 06:27:23.642731 15804 sys_catalog.cc:458] T 00000000000000000000000000000000 P f529a84bf69d4bbeb3377bf898056181 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:27:23.643391 15671 sys_catalog.cc:564] T 00000000000000000000000000000000 P f529a84bf69d4bbeb3377bf898056181 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:27:23.645372 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:27:23.639572 15467 master_runner.cc:186] Error getting master registration for 127.15.254.61:38597: OK, instance_id { permanent_uuid: "f529a84bf69d4bbeb3377bf898056181" instance_seqno: 1786084043351346 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 06:27:23.645849 15467 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 06:27:23.645646 15781 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaf3d026ea5494bb6bff733830f8360" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41285 } } peers { permanent_uuid: "f529a84bf69d4bbeb3377bf898056181" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 38597 } } peers { permanent_uuid: "805c291c3cdf48caac286f27afdd785c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } } }
I20260807 06:27:23.645920 15781 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:27:23.647375 15530 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ccaf3d026ea5494bb6bff733830f8360"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57112
Aug 07 06:27:23 dist-test-slave-44d2 krb5kdc[15449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084043, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.60@KRBTEST.COM for kudu/127.15.254.60@KRBTEST.COM
I20260807 06:27:23.647753 15530 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 06:27:23.647881 15530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:27:23.649762 15530 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaf3d026ea5494bb6bff733830f8360" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41285 } } peers { permanent_uuid: "f529a84bf69d4bbeb3377bf898056181" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 38597 } } peers { permanent_uuid: "805c291c3cdf48caac286f27afdd785c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } }
I20260807 06:27:23.651286 15629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ccaf3d026ea5494bb6bff733830f8360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f529a84bf69d4bbeb3377bf898056181"
I20260807 06:27:23.651518 15629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f529a84bf69d4bbeb3377bf898056181 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:27:23.651923 15530 leader_election.cc:288] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360 [CANDIDATE]: Term 1 election: Requested vote from peers f529a84bf69d4bbeb3377bf898056181 (127.15.254.61:38597), 805c291c3cdf48caac286f27afdd785c (127.15.254.60:33739)
W20260807 06:27:23.652593 15821 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f529a84bf69d4bbeb3377bf898056181: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:27:23.652666 15821 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:27:23.652969 15629 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f529a84bf69d4bbeb3377bf898056181 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccaf3d026ea5494bb6bff733830f8360 in term 1.
I20260807 06:27:23.653621 15501 leader_election.cc:302] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccaf3d026ea5494bb6bff733830f8360, f529a84bf69d4bbeb3377bf898056181; no voters:
I20260807 06:27:23.653923 15781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:27:23.654186 15781 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360 [term 1 LEADER]: Becoming Leader. State: Replica: ccaf3d026ea5494bb6bff733830f8360, State: Running, Role: LEADER
I20260807 06:27:23.654592 15781 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaf3d026ea5494bb6bff733830f8360" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41285 } } peers { permanent_uuid: "f529a84bf69d4bbeb3377bf898056181" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 38597 } } peers { permanent_uuid: "805c291c3cdf48caac286f27afdd785c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } }
I20260807 06:27:23.656855 15781 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccaf3d026ea5494bb6bff733830f8360. Latest consensus state: current_term: 1 leader_uuid: "ccaf3d026ea5494bb6bff733830f8360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaf3d026ea5494bb6bff733830f8360" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41285 } } peers { permanent_uuid: "f529a84bf69d4bbeb3377bf898056181" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 38597 } } peers { permanent_uuid: "805c291c3cdf48caac286f27afdd785c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } } }
I20260807 06:27:23.656965 15781 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360 [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:23.657219 15778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.633467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:33739 (local address 127.0.0.1:55030)
0807 06:27:23.633972 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:23.633986 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:27:23.634082 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:27:23.635790 (+ 1708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:27:23.635793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:27:23.635967 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:27:23.636061 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:23.636072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.643847 (+ 7775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.643850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:23.644832 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:23.644849 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.645475 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.645480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:23.645578 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:23.646155 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:27:23.646169 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:27:23.648630 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:27:23.655575 (+ 6945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:23.655579 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:23.655581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:23.655828 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:23.656128 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:23.656131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:23.656133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:23.656205 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:23.656671 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:27:23.656674 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:27:23.656766 (+ 92us) client_negotiation.cc:769] Sending connection context
0807 06:27:23.656864 (+ 98us) client_negotiation.cc:240] Negotiation successful
0807 06:27:23.657012 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":102,"threads_started":1}
I20260807 06:27:23.657895 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.633843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55030 (local address 127.15.254.60:33739)
0807 06:27:23.635588 (+ 1745us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:23.635592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:23.635607 (+ 15us) server_negotiation.cc:407] Connection header received
0807 06:27:23.635641 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:23.635643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:23.635683 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:23.636174 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:23.636248 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.643719 (+ 7471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.644998 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.645347 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.645846 (+ 499us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:23.649031 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:23.649036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:23.649039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:23.649066 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:23.655449 (+ 6383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.655955 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.655958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.655960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.656040 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.656291 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.656294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.656296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.656566 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:23.657501 (+ 935us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:23.657572 (+ 71us) server_negotiation.cc:299] Negotiation successful
0807 06:27:23.657714 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1674}
I20260807 06:27:23.661348 15771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 805c291c3cdf48caac286f27afdd785c: Bootstrap starting.
I20260807 06:27:23.661197 15578 master_runner.cc:186] Error getting master registration for 127.15.254.60:33739: OK, instance_id { permanent_uuid: "805c291c3cdf48caac286f27afdd785c" instance_seqno: 1786084043590223 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 06:27:23.665429 15578 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 06:27:23.665831 15572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.654198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57126 (local address 127.15.254.62:41285)
0807 06:27:23.654301 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:23.654309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:23.655773 (+ 1464us) server_negotiation.cc:407] Connection header received
0807 06:27:23.656063 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:23.656067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:23.656158 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:23.656219 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:23.657294 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.658076 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.659327 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.659677 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.659764 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:23.661750 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:23.661753 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:23.661756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:23.661781 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:23.663740 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.664296 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.664300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.664303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.664398 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.664744 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.664748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.664750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.664997 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:23.665119 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:23.665498 (+ 379us) server_negotiation.cc:299] Negotiation successful
0807 06:27:23.665632 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260807 06:27:23.667452 15771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 805c291c3cdf48caac286f27afdd785c: Neither blocks nor log segments found. Creating new log.
I20260807 06:27:23.668529 15771 log.cc:824] T 00000000000000000000000000000000 P 805c291c3cdf48caac286f27afdd785c: Log is configured to *not* fsync() on all Append() calls
I20260807 06:27:23.669503 15837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:27:23.670735 15837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:27:23.678541 15629 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f529a84bf69d4bbeb3377bf898056181 [term 1 FOLLOWER]: Refusing update from remote peer ccaf3d026ea5494bb6bff733830f8360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:27:23.679294 15831 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f529a84bf69d4bbeb3377bf898056181" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 38597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:27:23.679600 15771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 805c291c3cdf48caac286f27afdd785c: No bootstrap required, opened a new log
I20260807 06:27:23.682068 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.654834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60620 (local address 127.15.254.61:38597)
0807 06:27:23.654950 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:23.654955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:23.665866 (+ 10911us) server_negotiation.cc:407] Connection header received
0807 06:27:23.666064 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:23.666068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:23.666116 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:23.666167 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:23.672662 (+ 6495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.673551 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.674677 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.674998 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.675104 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:23.676991 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:23.676995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:23.676998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:23.677026 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:23.679830 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.680335 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.680338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.680340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.680450 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.680817 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.680820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.680822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.681035 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:23.681199 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:23.681781 (+ 582us) server_negotiation.cc:299] Negotiation successful
0807 06:27:23.681895 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260807 06:27:23.684456 15771 raft_consensus.cc:348] T 00000000000000000000000000000000 P 805c291c3cdf48caac286f27afdd785c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaf3d026ea5494bb6bff733830f8360" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41285 } } peers { permanent_uuid: "f529a84bf69d4bbeb3377bf898056181" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 38597 } } peers { permanent_uuid: "805c291c3cdf48caac286f27afdd785c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } }
I20260807 06:27:23.684754 15771 raft_consensus.cc:729] T 00000000000000000000000000000000 P 805c291c3cdf48caac286f27afdd785c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 805c291c3cdf48caac286f27afdd785c, State: Initialized, Role: FOLLOWER
I20260807 06:27:23.685089 15771 consensus_queue.cc:260] T 00000000000000000000000000000000 P 805c291c3cdf48caac286f27afdd785c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaf3d026ea5494bb6bff733830f8360" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41285 } } peers { permanent_uuid: "f529a84bf69d4bbeb3377bf898056181" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 38597 } } peers { permanent_uuid: "805c291c3cdf48caac286f27afdd785c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } }
I20260807 06:27:23.685815 15804 sys_catalog.cc:455] T 00000000000000000000000000000000 P f529a84bf69d4bbeb3377bf898056181 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccaf3d026ea5494bb6bff733830f8360. Latest consensus state: current_term: 1 leader_uuid: "ccaf3d026ea5494bb6bff733830f8360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaf3d026ea5494bb6bff733830f8360" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41285 } } peers { permanent_uuid: "f529a84bf69d4bbeb3377bf898056181" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 38597 } } peers { permanent_uuid: "805c291c3cdf48caac286f27afdd785c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } } }
I20260807 06:27:23.686108 15804 sys_catalog.cc:458] T 00000000000000000000000000000000 P f529a84bf69d4bbeb3377bf898056181 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:27:23.686080 15831 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ccaf3d026ea5494bb6bff733830f8360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaf3d026ea5494bb6bff733830f8360" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41285 } } peers { permanent_uuid: "f529a84bf69d4bbeb3377bf898056181" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 38597 } } peers { permanent_uuid: "805c291c3cdf48caac286f27afdd785c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } } }
I20260807 06:27:23.686255 15831 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360 [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:23.686419 15839 mvcc.cc:205] Tried to move back new op lower bound from 7315800242899238912 to 7315800242813251584. Current Snapshot: MvccSnapshot[applied={T|T < 7315800242899238912}]
I20260807 06:27:23.686576 15771 sys_catalog.cc:564] T 00000000000000000000000000000000 P 805c291c3cdf48caac286f27afdd785c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:27:23.686789 15733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 805c291c3cdf48caac286f27afdd785c [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:27:23.688519 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ccaf3d026ea5494bb6bff733830f8360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "805c291c3cdf48caac286f27afdd785c"
I20260807 06:27:23.688354 15733 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 805c291c3cdf48caac286f27afdd785c [term 1 FOLLOWER]: Refusing update from remote peer ccaf3d026ea5494bb6bff733830f8360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:27:23.689797 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:23.655480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55044 (local address 127.15.254.60:33739)
0807 06:27:23.655604 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:23.655610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:23.665766 (+ 10156us) server_negotiation.cc:407] Connection header received
0807 06:27:23.667299 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:23.667304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:23.667362 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:23.667429 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:23.673712 (+ 6283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.674526 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.681701 (+ 7175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:23.682042 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:23.682127 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:23.685729 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:23.685733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:23.685736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:23.685764 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:23.687729 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.688329 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.688333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.688335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.688437 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:23.688771 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:23.688774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:23.688776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:23.689017 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:23.689153 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:23.689517 (+ 364us) server_negotiation.cc:299] Negotiation successful
0807 06:27:23.689635 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 06:27:23.690196 15781 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "805c291c3cdf48caac286f27afdd785c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:27:23.692070 15737 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 805c291c3cdf48caac286f27afdd785c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccaf3d026ea5494bb6bff733830f8360 in term 1.
I20260807 06:27:23.692610 15837 catalog_manager.cc:1347] Generated new cluster ID: 8211493cca524317bf1cc359d211bf76
I20260807 06:27:23.692723 15837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:27:23.698446 15804 sys_catalog.cc:455] T 00000000000000000000000000000000 P f529a84bf69d4bbeb3377bf898056181 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ccaf3d026ea5494bb6bff733830f8360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaf3d026ea5494bb6bff733830f8360" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41285 } } peers { permanent_uuid: "f529a84bf69d4bbeb3377bf898056181" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 38597 } } peers { permanent_uuid: "805c291c3cdf48caac286f27afdd785c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } } }
I20260807 06:27:23.698591 15804 sys_catalog.cc:458] T 00000000000000000000000000000000 P f529a84bf69d4bbeb3377bf898056181 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:27:23.701445 15868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 805c291c3cdf48caac286f27afdd785c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccaf3d026ea5494bb6bff733830f8360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaf3d026ea5494bb6bff733830f8360" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41285 } } peers { permanent_uuid: "f529a84bf69d4bbeb3377bf898056181" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 38597 } } peers { permanent_uuid: "805c291c3cdf48caac286f27afdd785c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } } }
I20260807 06:27:23.701589 15868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 805c291c3cdf48caac286f27afdd785c [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:27:23.701680 15868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 805c291c3cdf48caac286f27afdd785c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccaf3d026ea5494bb6bff733830f8360. Latest consensus state: current_term: 1 leader_uuid: "ccaf3d026ea5494bb6bff733830f8360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaf3d026ea5494bb6bff733830f8360" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41285 } } peers { permanent_uuid: "f529a84bf69d4bbeb3377bf898056181" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 38597 } } peers { permanent_uuid: "805c291c3cdf48caac286f27afdd785c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } } }
I20260807 06:27:23.701778 15868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 805c291c3cdf48caac286f27afdd785c [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:27:23.701896 15868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 805c291c3cdf48caac286f27afdd785c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ccaf3d026ea5494bb6bff733830f8360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaf3d026ea5494bb6bff733830f8360" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41285 } } peers { permanent_uuid: "f529a84bf69d4bbeb3377bf898056181" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 38597 } } peers { permanent_uuid: "805c291c3cdf48caac286f27afdd785c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } } }
I20260807 06:27:23.702114 15868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 805c291c3cdf48caac286f27afdd785c [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:27:23.706519 15841 mvcc.cc:205] Tried to move back new op lower bound from 7315800242899238912 to 7315800242813251584. Current Snapshot: MvccSnapshot[applied={T|T < 7315800242899238912}]
I20260807 06:27:23.706719 15869 mvcc.cc:205] Tried to move back new op lower bound from 7315800242899238912 to 7315800242813251584. Current Snapshot: MvccSnapshot[applied={T|T < 7315800242899238912}]
I20260807 06:27:23.709184 15781 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ccaf3d026ea5494bb6bff733830f8360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaf3d026ea5494bb6bff733830f8360" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 41285 } } peers { permanent_uuid: "f529a84bf69d4bbeb3377bf898056181" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 38597 } } peers { permanent_uuid: "805c291c3cdf48caac286f27afdd785c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 33739 } } }
I20260807 06:27:23.709326 15781 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360 [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:23.712044 15837 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:27:23.713285 15837 catalog_manager.cc:1504] Loading token signing keys...
W20260807 06:27:23.716773 15874 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 805c291c3cdf48caac286f27afdd785c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:27:23.716867 15874 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:27:23.728894 15837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccaf3d026ea5494bb6bff733830f8360: Generated new TSK 0
I20260807 06:27:23.729905 15837 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 06:27:23.762837 15572 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:27:23.759384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57134 (local address 127.15.254.62:41285)
0807 06:27:23.759518 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:23.759522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:23.761421 (+ 1899us) server_negotiation.cc:407] Connection header received
0807 06:27:23.761846 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:23.761850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:23.761903 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:23.761957 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:23.762683 (+ 726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57134: BlockingRecv error: recv got EOF from 127.0.0.1:57134 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260807 06:27:23.764429 15678 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:27:23.760193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60626 (local address 127.15.254.61:38597)
0807 06:27:23.760310 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:23.760314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:23.761759 (+ 1445us) server_negotiation.cc:407] Connection header received
0807 06:27:23.763655 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:23.763659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:23.763711 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:23.763761 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:23.764306 (+ 545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60626: BlockingRecv error: recv got EOF from 127.0.0.1:60626 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260807 06:27:23.765697 15777 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:27:23.760826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55056 (local address 127.15.254.60:33739)
0807 06:27:23.760966 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:23.760970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:23.763467 (+ 2497us) server_negotiation.cc:407] Connection header received
0807 06:27:23.764933 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:23.764936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:23.764999 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:23.765049 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:23.765573 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55056: BlockingRecv error: recv got EOF from 127.0.0.1:55056 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260807 06:27:23.770713 15678 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:27:23.769160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60634 (local address 127.15.254.61:38597)
0807 06:27:23.769261 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:23.769265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:23.769722 (+ 457us) server_negotiation.cc:407] Connection header received
0807 06:27:23.769999 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:23.770004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:23.770084 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:23.770137 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:23.770621 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60634: BlockingRecv error: recv got EOF from 127.0.0.1:60634 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260807 06:27:23.775820 15678 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:27:23.774381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60638 (local address 127.15.254.61:38597)
0807 06:27:23.774476 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:23.774480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:23.774990 (+ 510us) server_negotiation.cc:407] Connection header received
0807 06:27:23.775171 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:23.775174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:23.775218 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:23.775266 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:23.775728 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60638: BlockingRecv error: recv got EOF from 127.0.0.1:60638 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260807 06:27:23.781296 15678 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:27:23.779735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60654 (local address 127.15.254.61:38597)
0807 06:27:23.779830 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:23.779834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:23.780325 (+ 491us) server_negotiation.cc:407] Connection header received
0807 06:27:23.780593 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:23.780597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:23.780644 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:23.780693 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:23.781208 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60654: BlockingRecv error: recv got EOF from 127.0.0.1:60654 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260807 06:27:23.789017 15678 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:27:23.787332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60658 (local address 127.15.254.61:38597)
0807 06:27:23.787464 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:23.787468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:23.787978 (+ 510us) server_negotiation.cc:407] Connection header received
0807 06:27:23.788286 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:23.788290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:23.788336 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:23.788385 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:23.788909 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60658: BlockingRecv error: recv got EOF from 127.0.0.1:60658 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260807 06:27:23.800755 15678 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:27:23.799151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60666 (local address 127.15.254.61:38597)
0807 06:27:23.799258 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:23.799262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:23.799783 (+ 521us) server_negotiation.cc:407] Connection header received
0807 06:27:23.799979 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:23.799983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:23.800032 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:23.800082 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:23.800647 (+ 565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60666: BlockingRecv error: recv got EOF from 127.0.0.1:60666 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260807 06:27:23.821375 15678 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:27:23.819414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.15.254.61:38597)
0807 06:27:23.819549 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:23.819553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:23.820216 (+ 663us) server_negotiation.cc:407] Connection header received
0807 06:27:23.820431 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:23.820438 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:23.820537 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:23.820606 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:23.821235 (+ 629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60668: BlockingRecv error: recv got EOF from 127.0.0.1:60668 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260807 06:27:23.858301 15678 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:27:23.856253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60682 (local address 127.15.254.61:38597)
0807 06:27:23.856401 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:23.856407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:23.857117 (+ 710us) server_negotiation.cc:407] Connection header received
0807 06:27:23.857362 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:23.857368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:23.857444 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:23.857511 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:23.858171 (+ 660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60682: BlockingRecv error: recv got EOF from 127.0.0.1:60682 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260807 06:27:23.926764 15678 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:27:23.925405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60694 (local address 127.15.254.61:38597)
0807 06:27:23.925571 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:23.925577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:23.925832 (+ 255us) server_negotiation.cc:407] Connection header received
0807 06:27:23.925955 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:23.925958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:23.926012 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:23.926120 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:23.926656 (+ 536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60694: BlockingRecv error: recv got EOF from 127.0.0.1:60694 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260807 06:27:24.058671 15678 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:27:24.057397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60702 (local address 127.15.254.61:38597)
0807 06:27:24.057539 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:24.057543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:24.057682 (+ 139us) server_negotiation.cc:407] Connection header received
0807 06:27:24.057766 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:24.057769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:24.057876 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:24.057975 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:24.058530 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60702: BlockingRecv error: recv got EOF from 127.0.0.1:60702 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260807 06:27:24.318631 15678 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:27:24.317229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60704 (local address 127.15.254.61:38597)
0807 06:27:24.317406 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:24.317411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:24.317578 (+ 167us) server_negotiation.cc:407] Connection header received
0807 06:27:24.317766 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:24.317769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:24.317821 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:24.317895 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:24.318486 (+ 591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60704: BlockingRecv error: recv got EOF from 127.0.0.1:60704 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 06:27:24.654474 15821 catalog_manager.cc:1259] Loaded cluster ID: 8211493cca524317bf1cc359d211bf76
I20260807 06:27:24.654564 15821 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f529a84bf69d4bbeb3377bf898056181: loading cluster ID for follower catalog manager: success
I20260807 06:27:24.656198 15821 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f529a84bf69d4bbeb3377bf898056181: acquiring CA information for follower catalog manager: success
I20260807 06:27:24.657464 15821 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f529a84bf69d4bbeb3377bf898056181: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 06:27:24.718334 15874 catalog_manager.cc:1259] Loaded cluster ID: 8211493cca524317bf1cc359d211bf76
I20260807 06:27:24.718434 15874 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 805c291c3cdf48caac286f27afdd785c: loading cluster ID for follower catalog manager: success
I20260807 06:27:24.720199 15874 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 805c291c3cdf48caac286f27afdd785c: acquiring CA information for follower catalog manager: success
I20260807 06:27:24.721422 15874 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 805c291c3cdf48caac286f27afdd785c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 06:27:24.837893 15972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:24.833114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60708 (local address 127.15.254.61:38597)
0807 06:27:24.833363 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:24.833367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:24.833383 (+ 16us) server_negotiation.cc:407] Connection header received
0807 06:27:24.833440 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:24.833443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:24.833573 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:24.833687 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:27:24.834157 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:24.834908 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:24.835998 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:24.836276 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:24.836375 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:24.836875 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:27:24.837422 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:27:24.837595 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:24.837642 (+ 47us) server_negotiation.cc:299] Negotiation successful
0807 06:27:24.837704 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":92,"threads_started":1}
I20260807 06:27:24.846587 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:24.841856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55062 (local address 127.15.254.60:33739)
0807 06:27:24.842148 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:24.842154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:24.842237 (+ 83us) server_negotiation.cc:407] Connection header received
0807 06:27:24.842481 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:24.842484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:24.842614 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:24.842761 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:27:24.843057 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:24.843731 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:24.844890 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:24.845146 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:24.845272 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:24.845662 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:27:24.846132 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:27:24.846287 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:24.846333 (+ 46us) server_negotiation.cc:299] Negotiation successful
0807 06:27:24.846406 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":92,"threads_started":1}
I20260807 06:27:24.855077 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:24.850886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55064 (local address 127.15.254.60:33739)
0807 06:27:24.851007 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:24.851012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:24.851373 (+ 361us) server_negotiation.cc:407] Connection header received
0807 06:27:24.851526 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:24.851529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:24.851580 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:24.851656 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:27:24.851970 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:24.852701 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:24.853687 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:24.854015 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:24.854125 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:24.854527 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:27:24.854613 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:27:24.854694 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:24.854821 (+ 127us) server_negotiation.cc:299] Negotiation successful
0807 06:27:24.854880 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 06:27:24.855099 15972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:24.850376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60710 (local address 127.15.254.61:38597)
0807 06:27:24.850535 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:24.850539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:24.850784 (+ 245us) server_negotiation.cc:407] Connection header received
0807 06:27:24.851021 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:24.851026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:24.851086 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:24.851139 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:27:24.851564 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:24.852290 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:24.853723 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:24.854081 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:24.854177 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:24.854568 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:27:24.854647 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:27:24.854734 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:24.854880 (+ 146us) server_negotiation.cc:299] Negotiation successful
0807 06:27:24.854940 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 06:27:24.855556 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:24.850056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57144 (local address 127.15.254.62:41285)
0807 06:27:24.850330 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:24.850335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:24.850354 (+ 19us) server_negotiation.cc:407] Connection header received
0807 06:27:24.850525 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:24.850528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:24.850753 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:24.850891 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:27:24.851321 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:24.852191 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:24.853720 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:24.854002 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:24.854136 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:24.854519 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:27:24.854908 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:27:24.855002 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:24.855328 (+ 326us) server_negotiation.cc:299] Negotiation successful
0807 06:27:24.855400 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":88,"threads_started":1}
I20260807 06:27:24.859437 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 15467
I20260807 06:27:24.872025 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 15578
I20260807 06:27:24.886057 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 15683
2026-08-07T06:27:24Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3325 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:27:24 dist-test-slave-44d2 krb5kdc[15999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:27:24 dist-test-slave-44d2 krb5kdc[15999](info): set up 2 sockets
Aug 07 06:27:24 dist-test-slave-44d2 krb5kdc[15999](info): commencing operation
krb5kdc: starting...
W20260807 06:27:26.143478 16376 mini_kdc.cc:121] Time spent starting KDC: real 1.228s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/krb5kdc/test-user.keytab.
Aug 07 06:27:26 dist-test-slave-44d2 krb5kdc[15999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:27:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:27:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.254.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
Entry for principal kudu/127.15.254.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab.
I20260807 06:27:26.284533 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:43019
--webserver_interface=127.15.254.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:43019,127.15.254.61:39745,127.15.254.60:46459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:27:26.402508 16016 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 06:27:26.402877 16016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:27:26.402958 16016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:27:26.403046 16016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:27:26.407128 16016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:27:26.407230 16016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:27:26.407275 16016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:27:26.407316 16016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:27:26.407354 16016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:27:26.412599 16016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:43019,127.15.254.61:39745,127.15.254.60:46459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.62.keytab
--principal=kudu/127.15.254.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.62:43019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.254.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:27:26.413672 16016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:27:26.414724 16016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:27:26.423806 16023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:26.423761 16028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:26.423826 16026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:27:26.424170 16016 server_base.cc:1034] running on GCE node
I20260807 06:27:26.424774 16016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:27:26.425904 16016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:27:26.427091 16016 hybrid_clock.cc:648] HybridClock initialized: now 1786084046427071 us; error 33 us; skew 500 ppm
Aug 07 06:27:26 dist-test-slave-44d2 krb5kdc[15999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084046, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:27:26.430142 16016 init.cc:377] Logged in from keytab as kudu/127.15.254.62@KRBTEST.COM (short username kudu)
I20260807 06:27:26.431126 16016 webserver.cc:460] Webserver started at http://127.15.254.62:36597/ using document root <none> and password file <none>
I20260807 06:27:26.431705 16016 fs_manager.cc:359] Metadata directory not provided
I20260807 06:27:26.431803 16016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:27:26.432050 16016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:27:26.434247 16016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-0/data/instance:
uuid: "d7c33dec402a408fb76d48cb009a7d46"
format_stamp: "Formatted at 2026-08-07 06:27:26 on dist-test-slave-44d2"
server_key: "b1ed8469327667c7880a89618a07a097"
server_key_iv: "78e655b110fe11c4fade85c2afa7a863"
server_key_version: "encryptionkey@0"
I20260807 06:27:26.434837 16016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-0/wal/instance:
uuid: "d7c33dec402a408fb76d48cb009a7d46"
format_stamp: "Formatted at 2026-08-07 06:27:26 on dist-test-slave-44d2"
server_key: "b1ed8469327667c7880a89618a07a097"
server_key_iv: "78e655b110fe11c4fade85c2afa7a863"
server_key_version: "encryptionkey@0"
I20260807 06:27:26.438833 16016 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260807 06:27:26.441576 16035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:27:26.443169 16016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:27:26.443336 16016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-0/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-0/wal
uuid: "d7c33dec402a408fb76d48cb009a7d46"
format_stamp: "Formatted at 2026-08-07 06:27:26 on dist-test-slave-44d2"
server_key: "b1ed8469327667c7880a89618a07a097"
server_key_iv: "78e655b110fe11c4fade85c2afa7a863"
server_key_version: "encryptionkey@0"
I20260807 06:27:26.443456 16016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:27:26.462332 16016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:27:26.465519 16016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:27:26.465728 16016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:27:26.473356 16016 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.62:43019
I20260807 06:27:26.473726 16110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.62:43019 every 8 connection(s)
I20260807 06:27:26.474414 16016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-0/data/info.pb
I20260807 06:27:26.477466 16112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:27:26.481422 16112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43019 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:27:26.482962 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 16016
I20260807 06:27:26.483148 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-0/wal/instance
I20260807 06:27:26.483479 16376 external_mini_cluster.cc:1428] Setting key 9bc7ae43185c4deda220a34ba02d8abd
Aug 07 06:27:26 dist-test-slave-44d2 krb5kdc[15999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084046, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.254.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.61@KRBTEST.COM" created.
I20260807 06:27:26.497267 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.484049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:43019 (local address 127.0.0.1:35736)
0807 06:27:26.484632 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.484679 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 06:27:26.485647 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:27:26.486372 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:27:26.486376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:27:26.486848 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:27:26.487105 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:26.487119 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.488272 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.488276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:26.489096 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:26.489106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.489764 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.489768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:26.489859 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:26.490503 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:27:26.490521 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:27:26.492030 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:27:26.494075 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:26.494085 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:26.494095 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:26.494336 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:26.494686 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:26.494689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:26.494691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:26.494810 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:26.495486 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:27:26.495493 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:27:26.495643 (+ 150us) client_negotiation.cc:769] Sending connection context
0807 06:27:26.495817 (+ 174us) client_negotiation.cc:240] Negotiation successful
0807 06:27:26.496234 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":116,"thread_start_us":153,"threads_started":1}
I20260807 06:27:26.497267 16115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.485062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35736 (local address 127.15.254.62:43019)
0807 06:27:26.485378 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:26.485389 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:26.485449 (+ 60us) server_negotiation.cc:407] Connection header received
0807 06:27:26.485960 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:26.485964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:26.486157 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:26.486374 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:26.487259 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.488143 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.489221 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.489607 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.489780 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:26.492182 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:26.492190 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:26.492204 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:26.492236 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:26.493943 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:26.494463 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:26.494474 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:26.494476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:26.494581 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:26.494905 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:26.494910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:26.494912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:26.495351 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:26.495517 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:26.495902 (+ 385us) server_negotiation.cc:299] Negotiation successful
0807 06:27:26.496183 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":215,"spinlock_wait_cycles":1920,"thread_start_us":99,"threads_started":1}
I20260807 06:27:26.499775 16112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.61" port: 39745 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:27:26.500656 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.500339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:39745 (local address 127.0.0.1:33480)
0807 06:27:26.500467 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.500591 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.61:39745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 06:27:26.500891 16039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.254.61:39745: connect: Connection refused (error 111)
I20260807 06:27:26.501721 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.501508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:39745 (local address 127.0.0.1:33490)
0807 06:27:26.501618 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.501669 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.61:39745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260807 06:27:26.501893 16112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.61:39745: Network error: Client connection negotiation failed: client connection to 127.15.254.61:39745: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.61.keytab.
Entry for principal kudu/127.15.254.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.61.keytab.
I20260807 06:27:26.515333 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.61:39745
--webserver_interface=127.15.254.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.61.keytab
--principal=kudu/127.15.254.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:43019,127.15.254.61:39745,127.15.254.60:46459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:27:26.551131 16112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.61" port: 39745 } attempt: 1
I20260807 06:27:26.552111 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.551782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:39745 (local address 127.0.0.1:33500)
0807 06:27:26.551958 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.552028 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.61:39745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260807 06:27:26.553018 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.552786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:39745 (local address 127.0.0.1:33502)
0807 06:27:26.552903 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.552950 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.61:39745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260807 06:27:26.553192 16112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.61:39745: Network error: Client connection negotiation failed: client connection to 127.15.254.61:39745: connect: Connection refused (error 111)
I20260807 06:27:26.621407 16112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.61" port: 39745 } attempt: 2
I20260807 06:27:26.622362 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.622040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:39745 (local address 127.0.0.1:33504)
0807 06:27:26.622202 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.622272 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.61:39745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 06:27:26.623136 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.622910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:39745 (local address 127.0.0.1:33516)
0807 06:27:26.623042 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.623085 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.61:39745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260807 06:27:26.623378 16112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.61:39745: Network error: Client connection negotiation failed: client connection to 127.15.254.61:39745: connect: Connection refused (error 111)
W20260807 06:27:26.634244 16119 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 06:27:26.634497 16119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:27:26.634560 16119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:27:26.634634 16119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:27:26.638654 16119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:27:26.638707 16119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:27:26.638726 16119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:27:26.638742 16119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:27:26.638783 16119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:27:26.644110 16119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:43019,127.15.254.61:39745,127.15.254.60:46459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.61.keytab
--principal=kudu/127.15.254.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.61:39745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.254.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:27:26.645215 16119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:27:26.646732 16119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:27:26.657842 16127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:26.657816 16129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:26.657816 16126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:27:26.658121 16119 server_base.cc:1034] running on GCE node
I20260807 06:27:26.658670 16119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:27:26.659822 16119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:27:26.661010 16119 hybrid_clock.cc:648] HybridClock initialized: now 1786084046660995 us; error 24 us; skew 500 ppm
Aug 07 06:27:26 dist-test-slave-44d2 krb5kdc[15999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084046, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:27:26.664033 16119 init.cc:377] Logged in from keytab as kudu/127.15.254.61@KRBTEST.COM (short username kudu)
I20260807 06:27:26.665033 16119 webserver.cc:460] Webserver started at http://127.15.254.61:35333/ using document root <none> and password file <none>
I20260807 06:27:26.665625 16119 fs_manager.cc:359] Metadata directory not provided
I20260807 06:27:26.665721 16119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:27:26.665973 16119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:27:26.668025 16119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-1/data/instance:
uuid: "21a8197c2b6a4818978ac05255328714"
format_stamp: "Formatted at 2026-08-07 06:27:26 on dist-test-slave-44d2"
server_key: "538ca41c5393edc426eb52d3db360c06"
server_key_iv: "0493c2c39748e3ab93bb377c0fad712b"
server_key_version: "encryptionkey@0"
I20260807 06:27:26.668665 16119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-1/wal/instance:
uuid: "21a8197c2b6a4818978ac05255328714"
format_stamp: "Formatted at 2026-08-07 06:27:26 on dist-test-slave-44d2"
server_key: "538ca41c5393edc426eb52d3db360c06"
server_key_iv: "0493c2c39748e3ab93bb377c0fad712b"
server_key_version: "encryptionkey@0"
I20260807 06:27:26.672423 16119 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:27:26.674978 16136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:27:26.676599 16119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 06:27:26.676759 16119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-1/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-1/wal
uuid: "21a8197c2b6a4818978ac05255328714"
format_stamp: "Formatted at 2026-08-07 06:27:26 on dist-test-slave-44d2"
server_key: "538ca41c5393edc426eb52d3db360c06"
server_key_iv: "0493c2c39748e3ab93bb377c0fad712b"
server_key_version: "encryptionkey@0"
I20260807 06:27:26.676890 16119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:27:26.688409 16119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:27:26.691586 16119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:27:26.691798 16119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:27:26.699491 16119 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.61:39745
I20260807 06:27:26.699796 16216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.61:39745 every 8 connection(s)
I20260807 06:27:26.700696 16119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-1/data/info.pb
I20260807 06:27:26.703522 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 16119
I20260807 06:27:26.703672 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-1/wal/instance
I20260807 06:27:26.704015 16376 external_mini_cluster.cc:1428] Setting key 79a68e3679b9c7ee0cc178f9f11c262c
I20260807 06:27:26.704072 16217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:27:26.708793 16217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43019 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.254.60@KRBTEST.COM; defaulting to no policy
I20260807 06:27:26.714597 16112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.61" port: 39745 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.254.60@KRBTEST.COM" created.
Aug 07 06:27:26 dist-test-slave-44d2 krb5kdc[15999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084046, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.61@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
Aug 07 06:27:26 dist-test-slave-44d2 krb5kdc[15999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084046, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for kudu/127.15.254.61@KRBTEST.COM
I20260807 06:27:26.726504 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.715196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:39745 (local address 127.0.0.1:33528)
0807 06:27:26.715396 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.715413 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:27:26.715492 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:27:26.716146 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:27:26.716150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:27:26.716353 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:27:26.716457 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:26.716463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.717902 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.717905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:26.718546 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:26.718554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.719316 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.719319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:26.719439 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:26.719942 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:27:26.719955 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:27:26.721614 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:27:26.724238 (+ 2624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:26.724245 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:26.724249 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:26.724583 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:26.725072 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:26.725078 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:26.725082 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:26.725169 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:26.726003 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:27:26.726007 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:27:26.726103 (+ 96us) client_negotiation.cc:769] Sending connection context
0807 06:27:26.726200 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 06:27:26.726333 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260807 06:27:26.727414 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.711331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:43019 (local address 127.0.0.1:35738)
0807 06:27:26.711949 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.712025 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0807 06:27:26.713347 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:27:26.714093 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:27:26.714100 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:27:26.714468 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:27:26.715245 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:26.715270 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.716293 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.716302 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:26.717617 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:26.717628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.718282 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.718289 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:26.718550 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:26.719308 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:27:26.719335 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:27:26.721810 (+ 2475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:27:26.724320 (+ 2510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:26.724334 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:26.724347 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:26.724705 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:26.725005 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:26.725009 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:26.725011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:26.725186 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:26.725612 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:27:26.725622 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:27:26.725824 (+ 202us) client_negotiation.cc:769] Sending connection context
0807 06:27:26.725999 (+ 175us) client_negotiation.cc:240] Negotiation successful
0807 06:27:26.726349 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":136,"threads_started":1}
I20260807 06:27:26.727422 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.715314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33528 (local address 127.15.254.61:39745)
0807 06:27:26.715705 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:26.715715 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:26.715735 (+ 20us) server_negotiation.cc:407] Connection header received
0807 06:27:26.715770 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:26.715774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:26.715984 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:26.716156 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:26.716619 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.717781 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.718676 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.719198 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.719314 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:26.721792 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:26.721803 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:26.721831 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:26.721866 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:26.724104 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:26.724723 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:26.724732 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:26.724736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:26.724903 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:26.725298 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:26.725304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:26.725308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:26.725866 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:26.726022 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:26.726353 (+ 331us) server_negotiation.cc:299] Negotiation successful
0807 06:27:26.726537 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":285,"spinlock_wait_cycles":1280,"thread_start_us":203,"threads_started":1}
I20260807 06:27:26.728125 16115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.710923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35738 (local address 127.15.254.62:43019)
0807 06:27:26.711115 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:26.711120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:26.712158 (+ 1038us) server_negotiation.cc:407] Connection header received
0807 06:27:26.713608 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:26.713611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:26.713666 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:26.713791 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:26.715427 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.716087 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.717755 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.718084 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.718212 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:26.721956 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:26.721960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:26.721962 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:26.721985 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:26.724160 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:26.724811 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:26.724814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:26.724816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:26.724903 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:26.725303 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:26.725306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:26.725307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:26.725487 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:26.725610 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:26.727908 (+ 2298us) server_negotiation.cc:299] Negotiation successful
0807 06:27:26.728022 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 06:27:26.729332 16217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.61" port: 39745 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:27:26.730204 16112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:27:26.730980 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.730696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46459 (local address 127.0.0.1:35764)
0807 06:27:26.730835 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.730897 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260807 06:27:26.731655 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.731420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46459 (local address 127.0.0.1:35768)
0807 06:27:26.731560 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.731604 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260807 06:27:26.731968 16112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.60:46459: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46459: connect: Connection refused (error 111)
Aug 07 06:27:26 dist-test-slave-44d2 krb5kdc[15999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084046, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.61@KRBTEST.COM for kudu/127.15.254.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.254.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.60.keytab.
Entry for principal kudu/127.15.254.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.60.keytab.
I20260807 06:27:26.738489 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.729924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:39745 (local address 127.0.0.1:33544)
0807 06:27:26.730066 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.730080 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:27:26.730175 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:27:26.730570 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:27:26.730572 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:27:26.730721 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:27:26.730855 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:26.730864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.731859 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.731863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:26.732456 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:26.732467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.733039 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.733042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:26.733184 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:26.733649 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:27:26.733660 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:27:26.735019 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:27:26.736850 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:26.736856 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:26.736858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:26.737100 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:26.737384 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:26.737387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:26.737388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:26.737470 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:26.737880 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:27:26.737883 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:27:26.737963 (+ 80us) client_negotiation.cc:769] Sending connection context
0807 06:27:26.738101 (+ 138us) client_negotiation.cc:240] Negotiation successful
0807 06:27:26.738239 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260807 06:27:26.738507 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.729956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33544 (local address 127.15.254.61:39745)
0807 06:27:26.730059 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:26.730063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:26.730161 (+ 98us) server_negotiation.cc:407] Connection header received
0807 06:27:26.730279 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:26.730282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:26.730330 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:26.730402 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:26.730992 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.731721 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.732670 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.732902 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.732989 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:26.735157 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:26.735161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:26.735162 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:26.735182 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:26.736747 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:26.737205 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:26.737209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:26.737211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:26.737287 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:26.737573 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:26.737576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:26.737578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:26.737781 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:26.737891 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:26.738209 (+ 318us) server_negotiation.cc:299] Negotiation successful
0807 06:27:26.738347 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260807 06:27:26.739308 16217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:27:26.739671 16376 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZSVWny/build/debug/bin/kudu
/tmp/dist-test-taskZSVWny/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.60:46459
--webserver_interface=127.15.254.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.60.keytab
--principal=kudu/127.15.254.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.254.20:33453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.254.62:43019,127.15.254.61:39745,127.15.254.60:46459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:27:26.740151 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.739765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46459 (local address 127.0.0.1:35782)
0807 06:27:26.740019 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.740097 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
W20260807 06:27:26.740785 16142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46459: connect: Connection refused (error 111)
I20260807 06:27:26.741863 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.741609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46459 (local address 127.0.0.1:35786)
0807 06:27:26.741739 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.741800 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 06:27:26.742045 16217 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.60:46459: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46459: connect: Connection refused (error 111)
I20260807 06:27:26.763166 16112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } attempt: 1
I20260807 06:27:26.764117 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.763735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46459 (local address 127.0.0.1:35796)
0807 06:27:26.763967 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.764031 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260807 06:27:26.764881 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.764654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46459 (local address 127.0.0.1:35810)
0807 06:27:26.764769 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.764818 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260807 06:27:26.765010 16112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.60:46459: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46459: connect: Connection refused (error 111)
I20260807 06:27:26.791298 16217 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } attempt: 1
I20260807 06:27:26.792301 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.791976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46459 (local address 127.0.0.1:35814)
0807 06:27:26.792147 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.792215 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260807 06:27:26.793190 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.792981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46459 (local address 127.0.0.1:35818)
0807 06:27:26.793083 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.793130 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260807 06:27:26.793346 16217 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.60:46459: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46459: connect: Connection refused (error 111)
I20260807 06:27:26.840207 16112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } attempt: 2
I20260807 06:27:26.841214 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.840839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46459 (local address 127.0.0.1:35834)
0807 06:27:26.841064 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.841130 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260807 06:27:26.841993 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.841763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46459 (local address 127.0.0.1:35844)
0807 06:27:26.841877 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.841938 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 06:27:26.842123 16112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.60:46459: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46459: connect: Connection refused (error 111)
W20260807 06:27:26.860642 16229 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 06:27:26.860992 16229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:27:26.861056 16229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:27:26.861135 16229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260807 06:27:26.861555 16217 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } attempt: 2
I20260807 06:27:26.862464 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.862153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46459 (local address 127.0.0.1:35852)
0807 06:27:26.862314 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.862384 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260807 06:27:26.863217 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.862996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46459 (local address 127.0.0.1:35868)
0807 06:27:26.863109 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.863162 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260807 06:27:26.863339 16217 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.254.60:46459: Network error: Client connection negotiation failed: client connection to 127.15.254.60:46459: connect: Connection refused (error 111)
W20260807 06:27:26.865815 16229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:27:26.865904 16229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:27:26.865967 16229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:27:26.866006 16229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:27:26.866044 16229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:27:26.871470 16229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.254.20:33453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.254.62:43019,127.15.254.61:39745,127.15.254.60:46459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/krb5kdc/kudu_127.15.254.60.keytab
--principal=kudu/127.15.254.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.254.60:46459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.254.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 05:39:39 UTC on ccycloud.quasar-xaluan.root.comops.site
build id 4571902
I20260807 06:27:26.872673 16229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:27:26.873725 16229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:27:26.883213 16241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:26.883134 16238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:27:26.883137 16237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:27:26.883533 16229 server_base.cc:1034] running on GCE node
I20260807 06:27:26.884102 16229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:27:26.885339 16229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:27:26.886569 16229 hybrid_clock.cc:648] HybridClock initialized: now 1786084046886549 us; error 40 us; skew 500 ppm
Aug 07 06:27:26 dist-test-slave-44d2 krb5kdc[15999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084046, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:27:26.889750 16229 init.cc:377] Logged in from keytab as kudu/127.15.254.60@KRBTEST.COM (short username kudu)
I20260807 06:27:26.890671 16229 webserver.cc:460] Webserver started at http://127.15.254.60:39281/ using document root <none> and password file <none>
I20260807 06:27:26.891259 16229 fs_manager.cc:359] Metadata directory not provided
I20260807 06:27:26.891393 16229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:27:26.891641 16229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:27:26.893822 16229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-2/data/instance:
uuid: "a0be382b54be4e45a5b161fe3a4d841c"
format_stamp: "Formatted at 2026-08-07 06:27:26 on dist-test-slave-44d2"
server_key: "0ef1b24d396806b65ff91fe1629a4aed"
server_key_iv: "aed4eba2c1309c3105a9d338c4a299b5"
server_key_version: "encryptionkey@0"
I20260807 06:27:26.894441 16229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-2/wal/instance:
uuid: "a0be382b54be4e45a5b161fe3a4d841c"
format_stamp: "Formatted at 2026-08-07 06:27:26 on dist-test-slave-44d2"
server_key: "0ef1b24d396806b65ff91fe1629a4aed"
server_key_iv: "aed4eba2c1309c3105a9d338c4a299b5"
server_key_version: "encryptionkey@0"
I20260807 06:27:26.898499 16229 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:27:26.901350 16255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:27:26.902704 16229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:27:26.902848 16229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-2/data,/tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-2/wal
uuid: "a0be382b54be4e45a5b161fe3a4d841c"
format_stamp: "Formatted at 2026-08-07 06:27:26 on dist-test-slave-44d2"
server_key: "0ef1b24d396806b65ff91fe1629a4aed"
server_key_iv: "aed4eba2c1309c3105a9d338c4a299b5"
server_key_version: "encryptionkey@0"
I20260807 06:27:26.903023 16229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:27:26.913300 16229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:27:26.916828 16229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:27:26.917058 16229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:27:26.924965 16229 rpc_server.cc:307] RPC server started. Bound to: 127.15.254.60:46459
I20260807 06:27:26.925398 16328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.254.60:46459 every 8 connection(s)
I20260807 06:27:26.926060 16229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-2/data/info.pb
I20260807 06:27:26.927973 16376 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZSVWny/build/debug/bin/kudu as pid 16229
I20260807 06:27:26.928088 16376 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZSVWny/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786083738959718-16376-0/minicluster-data/master-2/wal/instance
I20260807 06:27:26.928445 16376 external_mini_cluster.cc:1428] Setting key 24db986713422c9c75d335cb48b060c7
I20260807 06:27:26.929378 16329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:27:26.934509 16329 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43019 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:27:26 dist-test-slave-44d2 krb5kdc[15999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.60@KRBTEST.COM
I20260807 06:27:26.941288 16112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } attempt: 3
Aug 07 06:27:26 dist-test-slave-44d2 krb5kdc[15999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084046, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.60@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
Aug 07 06:27:26 dist-test-slave-44d2 krb5kdc[15999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084046, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.62@KRBTEST.COM for kudu/127.15.254.60@KRBTEST.COM
I20260807 06:27:26.947412 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.929979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35874 (local address 127.15.254.60:46459)
0807 06:27:26.930419 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:26.930445 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:26.930579 (+ 134us) server_negotiation.cc:407] Connection header received
0807 06:27:26.931531 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:26.931555 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:26.932011 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:26.932400 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:26.933250 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.934791 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.935840 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.936482 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.936689 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:26.939654 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:26.939661 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:26.939673 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:26.939706 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:26.942338 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:26.942900 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:26.942906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:26.942912 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:26.943035 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:26.943442 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:26.943446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:26.943448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:26.944007 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:26.944250 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:26.945762 (+ 1512us) server_negotiation.cc:299] Negotiation successful
0807 06:27:26.946064 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":113,"threads_started":1}
I20260807 06:27:26.948047 16115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.936606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35748 (local address 127.15.254.62:43019)
0807 06:27:26.936756 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:26.936761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:26.937150 (+ 389us) server_negotiation.cc:407] Connection header received
0807 06:27:26.937316 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:26.937320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:26.937389 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:26.937459 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:26.938401 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.938883 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.940436 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.940754 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.940951 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:26.943637 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:26.943641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:26.943645 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:26.943671 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:26.945366 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:26.946472 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:26.946475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:26.946477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:26.946568 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:26.946940 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:26.946944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:26.946946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:26.947184 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:26.947283 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:26.947717 (+ 434us) server_negotiation.cc:299] Negotiation successful
0807 06:27:26.947873 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 06:27:26.948473 16335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.936472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.62:43019 (local address 127.0.0.1:35748)
0807 06:27:26.937002 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.937053 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 06:27:26.937180 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:27:26.937570 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:27:26.937580 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:27:26.938069 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:27:26.938270 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:26.938279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.939032 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.939035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:26.939906 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:26.939918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.940913 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.940918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:26.941063 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:26.941642 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:27:26.941659 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:27:26.943480 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:27:26.945471 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:26.945523 (+ 52us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:26.945531 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:26.946349 (+ 818us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:26.946667 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:26.946671 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:26.946673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:26.946844 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:26.947340 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:27:26.947346 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:27:26.947489 (+ 143us) client_negotiation.cc:769] Sending connection context
0807 06:27:26.947644 (+ 155us) client_negotiation.cc:240] Negotiation successful
0807 06:27:26.947834 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":180,"threads_started":1}
I20260807 06:27:26.949759 16329 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.61" port: 39745 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:27:26.952502 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.941752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46459 (local address 127.0.0.1:35878)
0807 06:27:26.941984 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.942000 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:27:26.942086 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:27:26.942389 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:27:26.942396 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:27:26.942598 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:27:26.942711 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:26.942719 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.943743 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.943747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:26.944442 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:26.944475 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.945120 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.945123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:26.945188 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:26.945615 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:27:26.945629 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:27:26.947107 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:27:26.950778 (+ 3671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:26.950781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:26.950783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:26.951072 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:26.951426 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:26.951429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:26.951431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:26.951538 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:26.951972 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:27:26.951976 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:27:26.952077 (+ 101us) client_negotiation.cc:769] Sending connection context
0807 06:27:26.952184 (+ 107us) client_negotiation.cc:240] Negotiation successful
0807 06:27:26.952310 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260807 06:27:26.952648 16337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.941813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35878 (local address 127.15.254.60:46459)
0807 06:27:26.942142 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:26.942151 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:26.942165 (+ 14us) server_negotiation.cc:407] Connection header received
0807 06:27:26.942198 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:26.942201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:26.942245 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:26.942346 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:26.942856 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.943627 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.944621 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.944992 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.945111 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:26.947271 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:26.947275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:26.947277 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:26.947301 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:26.950451 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:26.951202 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:26.951206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:26.951209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:26.951308 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:26.951646 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:26.951650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:26.951651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:26.951860 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:26.951961 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:26.952269 (+ 308us) server_negotiation.cc:299] Negotiation successful
0807 06:27:26.952386 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":116,"threads_started":1}
I20260807 06:27:26.954507 16217 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } attempt: 3
Aug 07 06:27:26 dist-test-slave-44d2 krb5kdc[15999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.62@KRBTEST.COM
I20260807 06:27:26.956393 16112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46: Bootstrap starting.
Aug 07 06:27:26 dist-test-slave-44d2 krb5kdc[15999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084046, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.60@KRBTEST.COM for kudu/127.15.254.61@KRBTEST.COM
I20260807 06:27:26.959492 16112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46: Neither blocks nor log segments found. Creating new log.
I20260807 06:27:26.960541 16112 log.cc:824] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46: Log is configured to *not* fsync() on all Append() calls
Aug 07 06:27:26 dist-test-slave-44d2 krb5kdc[15999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084046, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.61@KRBTEST.COM for kudu/127.15.254.60@KRBTEST.COM
I20260807 06:27:26.962045 16115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.950437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35754 (local address 127.15.254.62:43019)
0807 06:27:26.950563 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:26.950567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:26.950651 (+ 84us) server_negotiation.cc:407] Connection header received
0807 06:27:26.950828 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:26.950834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:26.950887 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:26.950969 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:26.951875 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.952601 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.953641 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.953857 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.953970 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:26.957364 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:26.957368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:26.957371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:26.957397 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:26.959281 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:26.959934 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:26.959938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:26.959940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:26.960038 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:26.960376 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:26.960381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:26.960386 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:26.960793 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:26.961013 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:26.961724 (+ 711us) server_negotiation.cc:299] Negotiation successful
0807 06:27:26.961873 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 06:27:26.963002 16112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46: No bootstrap required, opened a new log
I20260807 06:27:26.963040 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.950632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33548 (local address 127.15.254.61:39745)
0807 06:27:26.950782 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:26.950789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:26.950805 (+ 16us) server_negotiation.cc:407] Connection header received
0807 06:27:26.950994 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:26.950997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:26.951056 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:26.951148 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:26.951634 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.952411 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.953856 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.954190 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.954269 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:26.958144 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:26.958150 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:26.958155 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:26.958189 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:26.960518 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:26.961024 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:26.961030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:26.961034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:26.961125 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:26.961836 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:26.961840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:26.961842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:26.962285 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:26.962364 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:26.962772 (+ 408us) server_negotiation.cc:299] Negotiation successful
0807 06:27:26.962890 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260807 06:27:26.963500 16335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.950355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.61:39745 (local address 127.0.0.1:33548)
0807 06:27:26.950625 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.950639 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:27:26.950867 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:27:26.951163 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:27:26.951165 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:27:26.951342 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:27:26.951492 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:26.951501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.952935 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.952939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:26.953705 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:26.953717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.954429 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.954432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:26.954524 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:26.955267 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:27:26.955280 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:27:26.957964 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:27:26.960654 (+ 2690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:26.960657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:26.960662 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:26.960911 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:26.961253 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:26.961259 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:26.961264 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:26.961435 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:26.962460 (+ 1025us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:27:26.962462 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:27:26.962562 (+ 100us) client_negotiation.cc:769] Sending connection context
0807 06:27:26.963176 (+ 614us) client_negotiation.cc:240] Negotiation successful
0807 06:27:26.963320 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"spinlock_wait_cycles":896}
I20260807 06:27:26.965165 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.954943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46459 (local address 127.0.0.1:35884)
0807 06:27:26.955102 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.955121 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 06:27:26.955213 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:27:26.956182 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:27:26.956185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:27:26.956374 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:27:26.956520 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:26.956528 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.957556 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.957559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:26.958357 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:26.958369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.958958 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.958962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:26.959043 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:26.959482 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:27:26.959497 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:27:26.961057 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:27:26.963334 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:26.963339 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:26.963342 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:26.963594 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:26.964074 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:26.964077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:26.964079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:26.964169 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:26.964669 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:27:26.964672 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:27:26.964763 (+ 91us) client_negotiation.cc:769] Sending connection context
0807 06:27:26.964844 (+ 81us) client_negotiation.cc:240] Negotiation successful
0807 06:27:26.965004 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260807 06:27:26.965121 16329 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:27:26.966450 16337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.955176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35884 (local address 127.15.254.60:46459)
0807 06:27:26.955922 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:26.955926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:26.955939 (+ 13us) server_negotiation.cc:407] Connection header received
0807 06:27:26.955972 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:26.955974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:26.956014 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:26.956638 (+ 624us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:26.956724 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.957416 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.958511 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.958810 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.959272 (+ 462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:26.961232 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:26.961236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:26.961238 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:26.961265 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:26.962975 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:26.963851 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:26.963854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:26.963855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:26.963955 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:26.964287 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:26.964290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:26.964292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:26.964552 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:26.965628 (+ 1076us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:26.966122 (+ 494us) server_negotiation.cc:299] Negotiation successful
0807 06:27:26.966305 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658}
I20260807 06:27:26.967758 16112 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c33dec402a408fb76d48cb009a7d46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43019 } } peers { permanent_uuid: "21a8197c2b6a4818978ac05255328714" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 39745 } } peers { permanent_uuid: "a0be382b54be4e45a5b161fe3a4d841c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } }
I20260807 06:27:26.967988 16112 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c33dec402a408fb76d48cb009a7d46, State: Initialized, Role: FOLLOWER
I20260807 06:27:26.968351 16112 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c33dec402a408fb76d48cb009a7d46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43019 } } peers { permanent_uuid: "21a8197c2b6a4818978ac05255328714" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 39745 } } peers { permanent_uuid: "a0be382b54be4e45a5b161fe3a4d841c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } }
I20260807 06:27:26.968935 16339 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c33dec402a408fb76d48cb009a7d46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43019 } } peers { permanent_uuid: "21a8197c2b6a4818978ac05255328714" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 39745 } } peers { permanent_uuid: "a0be382b54be4e45a5b161fe3a4d841c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } } }
I20260807 06:27:26.969110 16339 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:27:26.969870 16112 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:27:26.970497 16217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21a8197c2b6a4818978ac05255328714: Bootstrap starting.
Aug 07 06:27:26 dist-test-slave-44d2 krb5kdc[15999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084046, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.254.60@KRBTEST.COM for kudu/127.15.254.60@KRBTEST.COM
I20260807 06:27:26.974747 16335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.965530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.254.60:46459 (local address 127.0.0.1:35886)
0807 06:27:26.965742 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0807 06:27:26.965754 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 06:27:26.965817 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:27:26.966059 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:27:26.966062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:27:26.966297 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:27:26.966380 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:26.966387 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.967392 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.967395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:26.968144 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:27:26.968155 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.968951 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.968954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:27:26.969034 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:26.969558 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:27:26.969573 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:27:26.971178 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:27:26.973261 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:26.973267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:26.973270 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:26.973470 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:26.973755 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:27:26.973759 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:27:26.973761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:27:26.973843 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:27:26.974210 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:27:26.974213 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:27:26.974295 (+ 82us) client_negotiation.cc:769] Sending connection context
0807 06:27:26.974414 (+ 119us) client_negotiation.cc:240] Negotiation successful
0807 06:27:26.974559 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
Aug 07 06:27:26 dist-test-slave-44d2 krb5kdc[15999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786084046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.254.61@KRBTEST.COM
I20260807 06:27:26.975227 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.965728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35886 (local address 127.15.254.60:46459)
0807 06:27:26.965836 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:26.965840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:26.965852 (+ 12us) server_negotiation.cc:407] Connection header received
0807 06:27:26.965898 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:26.965900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:26.965940 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:26.966019 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:26.966495 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.967235 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.968274 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.968791 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.969326 (+ 535us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:26.971328 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:26.971331 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:26.971334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:26.971356 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:26.973141 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:26.973580 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:26.973583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:26.973585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:26.973664 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:26.973941 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:26.973944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:26.973946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:26.974115 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:26.974221 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:26.974945 (+ 724us) server_negotiation.cc:299] Negotiation successful
0807 06:27:26.975070 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260807 06:27:26.978101 16356 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:27:26.978189 16356 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:27:26.978412 16329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0be382b54be4e45a5b161fe3a4d841c: Bootstrap starting.
I20260807 06:27:26.979950 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:26.963398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33554 (local address 127.15.254.61:39745)
0807 06:27:26.963497 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:26.963500 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:26.963891 (+ 391us) server_negotiation.cc:407] Connection header received
0807 06:27:26.964036 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:26.964039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:26.964078 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:26.964175 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:26.964910 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.965559 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.966648 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:26.966983 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:26.967060 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:26.976551 (+ 9491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:26.976554 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:26.976557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:26.976591 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:26.978026 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:26.978640 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:26.978647 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:26.978653 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:26.978734 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:26.979036 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:26.979040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:26.979042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:26.979244 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:26.979443 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:26.979661 (+ 218us) server_negotiation.cc:299] Negotiation successful
0807 06:27:26.979784 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260807 06:27:26.981107 16329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0be382b54be4e45a5b161fe3a4d841c: Neither blocks nor log segments found. Creating new log.
I20260807 06:27:26.982069 16329 log.cc:824] T 00000000000000000000000000000000 P a0be382b54be4e45a5b161fe3a4d841c: Log is configured to *not* fsync() on all Append() calls
I20260807 06:27:26.980577 16016 master_runner.cc:186] Error getting master registration for 127.15.254.61:39745: OK, instance_id { permanent_uuid: "21a8197c2b6a4818978ac05255328714" instance_seqno: 1786084046697278 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 06:27:26.984302 16016 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 06:27:26.984375 16376 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:27:26.984380 16217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21a8197c2b6a4818978ac05255328714: Neither blocks nor log segments found. Creating new log.
I20260807 06:27:26.985409 16217 log.cc:824] T 00000000000000000000000000000000 P 21a8197c2b6a4818978ac05255328714: Log is configured to *not* fsync() on all Append() calls
I20260807 06:27:26.985601 16329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0be382b54be4e45a5b161fe3a4d841c: No bootstrap required, opened a new log
I20260807 06:27:26.985634 16076 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d7c33dec402a408fb76d48cb009a7d46"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35754
I20260807 06:27:26.985836 16076 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 06:27:26.985916 16076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:27:26.986965 16076 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c33dec402a408fb76d48cb009a7d46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43019 } } peers { permanent_uuid: "21a8197c2b6a4818978ac05255328714" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 39745 } } peers { permanent_uuid: "a0be382b54be4e45a5b161fe3a4d841c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } }
I20260807 06:27:26.987884 16076 leader_election.cc:288] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46 [CANDIDATE]: Term 1 election: Requested vote from peers 21a8197c2b6a4818978ac05255328714 (127.15.254.61:39745), a0be382b54be4e45a5b161fe3a4d841c (127.15.254.60:46459)
I20260807 06:27:26.990689 16217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21a8197c2b6a4818978ac05255328714: No bootstrap required, opened a new log
I20260807 06:27:26.990989 16329 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0be382b54be4e45a5b161fe3a4d841c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c33dec402a408fb76d48cb009a7d46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43019 } } peers { permanent_uuid: "21a8197c2b6a4818978ac05255328714" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 39745 } } peers { permanent_uuid: "a0be382b54be4e45a5b161fe3a4d841c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } }
I20260807 06:27:26.991269 16329 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0be382b54be4e45a5b161fe3a4d841c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0be382b54be4e45a5b161fe3a4d841c, State: Initialized, Role: FOLLOWER
I20260807 06:27:26.991652 16329 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0be382b54be4e45a5b161fe3a4d841c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c33dec402a408fb76d48cb009a7d46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43019 } } peers { permanent_uuid: "21a8197c2b6a4818978ac05255328714" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 39745 } } peers { permanent_uuid: "a0be382b54be4e45a5b161fe3a4d841c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } }
I20260807 06:27:26.992347 16367 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0be382b54be4e45a5b161fe3a4d841c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c33dec402a408fb76d48cb009a7d46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43019 } } peers { permanent_uuid: "21a8197c2b6a4818978ac05255328714" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 39745 } } peers { permanent_uuid: "a0be382b54be4e45a5b161fe3a4d841c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } } }
I20260807 06:27:26.992789 16367 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0be382b54be4e45a5b161fe3a4d841c [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:27:26.993152 16329 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0be382b54be4e45a5b161fe3a4d841c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:27:26.993232 16289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7c33dec402a408fb76d48cb009a7d46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a0be382b54be4e45a5b161fe3a4d841c"
I20260807 06:27:26.993451 16289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0be382b54be4e45a5b161fe3a4d841c [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:27:26.995270 16217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21a8197c2b6a4818978ac05255328714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c33dec402a408fb76d48cb009a7d46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43019 } } peers { permanent_uuid: "21a8197c2b6a4818978ac05255328714" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 39745 } } peers { permanent_uuid: "a0be382b54be4e45a5b161fe3a4d841c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } }
I20260807 06:27:26.995512 16217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21a8197c2b6a4818978ac05255328714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21a8197c2b6a4818978ac05255328714, State: Initialized, Role: FOLLOWER
I20260807 06:27:26.995857 16217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21a8197c2b6a4818978ac05255328714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c33dec402a408fb76d48cb009a7d46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43019 } } peers { permanent_uuid: "21a8197c2b6a4818978ac05255328714" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 39745 } } peers { permanent_uuid: "a0be382b54be4e45a5b161fe3a4d841c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } }
I20260807 06:27:26.996887 16289 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a0be382b54be4e45a5b161fe3a4d841c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c33dec402a408fb76d48cb009a7d46 in term 1.
I20260807 06:27:26.997344 16217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21a8197c2b6a4818978ac05255328714 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:27:26.997502 16040 leader_election.cc:302] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0be382b54be4e45a5b161fe3a4d841c, d7c33dec402a408fb76d48cb009a7d46; no voters:
I20260807 06:27:26.997841 16339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:27:26.997999 16176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7c33dec402a408fb76d48cb009a7d46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "21a8197c2b6a4818978ac05255328714"
I20260807 06:27:26.998138 16339 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46 [term 1 LEADER]: Becoming Leader. State: Replica: d7c33dec402a408fb76d48cb009a7d46, State: Running, Role: LEADER
I20260807 06:27:26.998270 16176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21a8197c2b6a4818978ac05255328714 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:27:26.998540 16339 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c33dec402a408fb76d48cb009a7d46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43019 } } peers { permanent_uuid: "21a8197c2b6a4818978ac05255328714" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 39745 } } peers { permanent_uuid: "a0be382b54be4e45a5b161fe3a4d841c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } }
I20260807 06:27:27.000079 16176 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 21a8197c2b6a4818978ac05255328714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c33dec402a408fb76d48cb009a7d46 in term 1.
I20260807 06:27:27.000886 16339 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7c33dec402a408fb76d48cb009a7d46. Latest consensus state: current_term: 1 leader_uuid: "d7c33dec402a408fb76d48cb009a7d46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c33dec402a408fb76d48cb009a7d46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43019 } } peers { permanent_uuid: "21a8197c2b6a4818978ac05255328714" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 39745 } } peers { permanent_uuid: "a0be382b54be4e45a5b161fe3a4d841c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } } }
I20260807 06:27:27.001027 16339 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46 [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:27.003155 16368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21a8197c2b6a4818978ac05255328714 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c33dec402a408fb76d48cb009a7d46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43019 } } peers { permanent_uuid: "21a8197c2b6a4818978ac05255328714" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 39745 } } peers { permanent_uuid: "a0be382b54be4e45a5b161fe3a4d841c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } } }
I20260807 06:27:27.003304 16368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21a8197c2b6a4818978ac05255328714 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:27:27.004326 16412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:27:27.005483 16412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260807 06:27:27.006198 16417 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 21a8197c2b6a4818978ac05255328714: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:27:27.006373 16417 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260807 06:27:27.010394 16408 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0be382b54be4e45a5b161fe3a4d841c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:27:27.010480 16408 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:27:27.019194 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:27.001153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33570 (local address 127.15.254.61:39745)
0807 06:27:27.001302 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:27.001308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:27.008076 (+ 6768us) server_negotiation.cc:407] Connection header received
0807 06:27:27.008115 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:27.008118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:27.008164 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:27.008237 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:27.009615 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:27.010424 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:27.012213 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:27.012608 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:27.012692 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:27.014887 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:27.014891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:27.014894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:27.014926 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:27.016938 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:27.017481 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:27.017485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:27.017487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:27.017598 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:27.017960 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:27.017964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:27.017966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:27.018204 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:27.018320 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:27.018860 (+ 540us) server_negotiation.cc:299] Negotiation successful
0807 06:27:27.019014 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 06:27:27.025770 16176 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 21a8197c2b6a4818978ac05255328714 [term 1 FOLLOWER]: Refusing update from remote peer d7c33dec402a408fb76d48cb009a7d46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:27:27.025810 16289 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a0be382b54be4e45a5b161fe3a4d841c [term 1 FOLLOWER]: Refusing update from remote peer d7c33dec402a408fb76d48cb009a7d46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:27:27.026539 16369 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21a8197c2b6a4818978ac05255328714" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 39745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:27:27.027449 16369 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0be382b54be4e45a5b161fe3a4d841c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:27:27.029281 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:27.002389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35896 (local address 127.15.254.60:46459)
0807 06:27:27.002508 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:27.002512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:27.011478 (+ 8966us) server_negotiation.cc:407] Connection header received
0807 06:27:27.011576 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:27.011579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:27.011627 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:27.011712 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:27.012781 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:27.013566 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:27.014733 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:27.015103 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:27.015179 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:27.023540 (+ 8361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:27.023546 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:27.023549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:27.023584 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:27.027220 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:27.027759 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:27.027766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:27.027768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:27.027868 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:27.028215 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:27.028219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:27.028221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:27.028456 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:27.028606 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:27.028995 (+ 389us) server_negotiation.cc:299] Negotiation successful
0807 06:27:27.029129 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 06:27:27.031641 16368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21a8197c2b6a4818978ac05255328714 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7c33dec402a408fb76d48cb009a7d46. Latest consensus state: current_term: 1 leader_uuid: "d7c33dec402a408fb76d48cb009a7d46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c33dec402a408fb76d48cb009a7d46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43019 } } peers { permanent_uuid: "21a8197c2b6a4818978ac05255328714" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 39745 } } peers { permanent_uuid: "a0be382b54be4e45a5b161fe3a4d841c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } } }
I20260807 06:27:27.031797 16368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21a8197c2b6a4818978ac05255328714 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:27:27.036091 16367 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0be382b54be4e45a5b161fe3a4d841c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7c33dec402a408fb76d48cb009a7d46. Latest consensus state: current_term: 1 leader_uuid: "d7c33dec402a408fb76d48cb009a7d46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c33dec402a408fb76d48cb009a7d46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43019 } } peers { permanent_uuid: "21a8197c2b6a4818978ac05255328714" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 39745 } } peers { permanent_uuid: "a0be382b54be4e45a5b161fe3a4d841c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } } }
I20260807 06:27:27.036228 16367 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0be382b54be4e45a5b161fe3a4d841c [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:27:27.037161 16115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:27.002596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35758 (local address 127.15.254.62:43019)
0807 06:27:27.002713 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:27.002716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:27.003928 (+ 1212us) server_negotiation.cc:407] Connection header received
0807 06:27:27.004070 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:27.004073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:27.004124 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:27.004211 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:27.008040 (+ 3829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:27.010610 (+ 2570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:27.024043 (+ 13433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:27.024436 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:27.024555 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:27.033037 (+ 8482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:27:27.033041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:27:27.033044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:27:27.033075 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:27:27.035094 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:27.035609 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:27.035613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:27.035615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:27.035709 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:27:27.036076 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:27:27.036080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:27:27.036082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:27:27.036322 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:27:27.036458 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:27.036849 (+ 391us) server_negotiation.cc:299] Negotiation successful
0807 06:27:27.036999 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260807 06:27:27.043234 16369 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d7c33dec402a408fb76d48cb009a7d46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c33dec402a408fb76d48cb009a7d46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43019 } } peers { permanent_uuid: "21a8197c2b6a4818978ac05255328714" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 39745 } } peers { permanent_uuid: "a0be382b54be4e45a5b161fe3a4d841c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } } }
I20260807 06:27:27.043373 16369 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46 [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:27.043526 16423 mvcc.cc:205] Tried to move back new op lower bound from 7315800256587935744 to 7315800256510373888. Current Snapshot: MvccSnapshot[applied={T|T < 7315800256587935744}]
I20260807 06:27:27.044674 16367 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0be382b54be4e45a5b161fe3a4d841c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d7c33dec402a408fb76d48cb009a7d46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c33dec402a408fb76d48cb009a7d46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43019 } } peers { permanent_uuid: "21a8197c2b6a4818978ac05255328714" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 39745 } } peers { permanent_uuid: "a0be382b54be4e45a5b161fe3a4d841c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } } }
I20260807 06:27:27.044804 16367 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0be382b54be4e45a5b161fe3a4d841c [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:27:27.048702 16339 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d7c33dec402a408fb76d48cb009a7d46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c33dec402a408fb76d48cb009a7d46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43019 } } peers { permanent_uuid: "21a8197c2b6a4818978ac05255328714" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 39745 } } peers { permanent_uuid: "a0be382b54be4e45a5b161fe3a4d841c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } } }
I20260807 06:27:27.048825 16339 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46 [sys.catalog]: This master's current role is: LEADER
I20260807 06:27:27.049783 16368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21a8197c2b6a4818978ac05255328714 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d7c33dec402a408fb76d48cb009a7d46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c33dec402a408fb76d48cb009a7d46" member_type: VOTER last_known_addr { host: "127.15.254.62" port: 43019 } } peers { permanent_uuid: "21a8197c2b6a4818978ac05255328714" member_type: VOTER last_known_addr { host: "127.15.254.61" port: 39745 } } peers { permanent_uuid: "a0be382b54be4e45a5b161fe3a4d841c" member_type: VOTER last_known_addr { host: "127.15.254.60" port: 46459 } } }
I20260807 06:27:27.049906 16368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21a8197c2b6a4818978ac05255328714 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:27:27.050529 16424 mvcc.cc:205] Tried to move back new op lower bound from 7315800256587935744 to 7315800256510373888. Current Snapshot: MvccSnapshot[applied={T|T < 7315800256587935744}]
I20260807 06:27:27.050897 16427 mvcc.cc:205] Tried to move back new op lower bound from 7315800256587935744 to 7315800256510373888. Current Snapshot: MvccSnapshot[applied={T|T < 7315800256587935744}]
I20260807 06:27:27.053098 16412 catalog_manager.cc:1347] Generated new cluster ID: 508a4ff25d75424e854fd4fee6b4143a
I20260807 06:27:27.053164 16412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:27:27.082113 16412 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:27:27.086040 16412 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:27:27.098035 16412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7c33dec402a408fb76d48cb009a7d46: Generated new TSK 0
I20260807 06:27:27.098995 16412 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 06:27:27.132769 16334 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:27:27.129412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35904 (local address 127.15.254.60:46459)
0807 06:27:27.129556 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:27.129561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:27.131545 (+ 1984us) server_negotiation.cc:407] Connection header received
0807 06:27:27.131749 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:27.131753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:27.131805 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:27.131859 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:27.132608 (+ 749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35904: BlockingRecv error: recv got EOF from 127.0.0.1:35904 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260807 06:27:27.134274 16223 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:27:27.130243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33586 (local address 127.15.254.61:39745)
0807 06:27:27.130359 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:27.130364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:27.133263 (+ 2899us) server_negotiation.cc:407] Connection header received
0807 06:27:27.133505 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:27.133510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:27.133567 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:27.133620 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:27.134147 (+ 527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33586: BlockingRecv error: recv got EOF from 127.0.0.1:33586 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260807 06:27:27.135807 16115 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:27:27.130924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35766 (local address 127.15.254.62:43019)
0807 06:27:27.131043 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:27.131047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:27.134803 (+ 3756us) server_negotiation.cc:407] Connection header received
0807 06:27:27.135038 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:27.135041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:27.135093 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:27.135145 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:27.135689 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35766: BlockingRecv error: recv got EOF from 127.0.0.1:35766 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260807 06:27:27.140300 16223 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:27:27.138773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33590 (local address 127.15.254.61:39745)
0807 06:27:27.138881 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:27.138885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:27.139349 (+ 464us) server_negotiation.cc:407] Connection header received
0807 06:27:27.139580 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:27.139584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:27.139659 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:27.139710 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:27.140202 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33590: BlockingRecv error: recv got EOF from 127.0.0.1:33590 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260807 06:27:27.145412 16223 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:27:27.143914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33598 (local address 127.15.254.61:39745)
0807 06:27:27.144014 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:27.144018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:27.144533 (+ 515us) server_negotiation.cc:407] Connection header received
0807 06:27:27.144728 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:27.144731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:27.144782 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:27.144834 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:27.145315 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33598: BlockingRecv error: recv got EOF from 127.0.0.1:33598 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260807 06:27:27.151048 16223 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:27:27.149613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33614 (local address 127.15.254.61:39745)
0807 06:27:27.149711 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:27.149715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:27.150180 (+ 465us) server_negotiation.cc:407] Connection header received
0807 06:27:27.150358 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:27.150361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:27.150411 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:27.150474 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:27.150952 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33614: BlockingRecv error: recv got EOF from 127.0.0.1:33614 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260807 06:27:27.158728 16223 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:27:27.157214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33618 (local address 127.15.254.61:39745)
0807 06:27:27.157345 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:27.157349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:27.157843 (+ 494us) server_negotiation.cc:407] Connection header received
0807 06:27:27.158026 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:27.158029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:27.158077 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:27.158128 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:27.158626 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33618: BlockingRecv error: recv got EOF from 127.0.0.1:33618 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260807 06:27:27.170972 16223 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:27:27.169177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33622 (local address 127.15.254.61:39745)
0807 06:27:27.169293 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:27.169298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:27.169869 (+ 571us) server_negotiation.cc:407] Connection header received
0807 06:27:27.170090 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:27.170094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:27.170149 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:27.170224 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:27.170863 (+ 639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33622: BlockingRecv error: recv got EOF from 127.0.0.1:33622 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260807 06:27:27.191922 16223 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:27:27.189922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33626 (local address 127.15.254.61:39745)
0807 06:27:27.190072 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:27.190077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:27.190751 (+ 674us) server_negotiation.cc:407] Connection header received
0807 06:27:27.191007 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:27.191011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:27.191072 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:27.191130 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:27.191773 (+ 643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33626: BlockingRecv error: recv got EOF from 127.0.0.1:33626 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260807 06:27:27.228166 16223 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:27:27.226822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33630 (local address 127.15.254.61:39745)
0807 06:27:27.226954 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:27.226959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:27.227285 (+ 326us) server_negotiation.cc:407] Connection header received
0807 06:27:27.227387 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:27.227393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:27.227476 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:27.227592 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:27.228042 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33630: BlockingRecv error: recv got EOF from 127.0.0.1:33630 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260807 06:27:27.296272 16223 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:27:27.294885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33634 (local address 127.15.254.61:39745)
0807 06:27:27.295086 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:27.295092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:27.295310 (+ 218us) server_negotiation.cc:407] Connection header received
0807 06:27:27.295432 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:27.295435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:27.295490 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:27.295584 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:27.296119 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33634: BlockingRecv error: recv got EOF from 127.0.0.1:33634 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260807 06:27:27.427851 16223 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:27:27.426571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33640 (local address 127.15.254.61:39745)
0807 06:27:27.426745 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:27.426749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:27.426881 (+ 132us) server_negotiation.cc:407] Connection header received
0807 06:27:27.427097 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:27.427100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:27.427157 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:27.427244 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:27.427743 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33640: BlockingRecv error: recv got EOF from 127.0.0.1:33640 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260807 06:27:27.687379 16223 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:27:27.686271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33656 (local address 127.15.254.61:39745)
0807 06:27:27.686448 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:27.686451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:27.686521 (+ 70us) server_negotiation.cc:407] Connection header received
0807 06:27:27.686649 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:27.686651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:27.686702 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:27.686783 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:27:27.687267 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33656: BlockingRecv error: recv got EOF from 127.0.0.1:33656 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 06:27:28.008100 16417 catalog_manager.cc:1259] Loaded cluster ID: 508a4ff25d75424e854fd4fee6b4143a
I20260807 06:27:28.008193 16417 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 21a8197c2b6a4818978ac05255328714: loading cluster ID for follower catalog manager: success
I20260807 06:27:28.010075 16417 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 21a8197c2b6a4818978ac05255328714: acquiring CA information for follower catalog manager: success
I20260807 06:27:28.011343 16417 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 21a8197c2b6a4818978ac05255328714: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 06:27:28.011781 16408 catalog_manager.cc:1259] Loaded cluster ID: 508a4ff25d75424e854fd4fee6b4143a
I20260807 06:27:28.011876 16408 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a0be382b54be4e45a5b161fe3a4d841c: loading cluster ID for follower catalog manager: success
I20260807 06:27:28.013602 16408 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a0be382b54be4e45a5b161fe3a4d841c: acquiring CA information for follower catalog manager: success
I20260807 06:27:28.014793 16408 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a0be382b54be4e45a5b161fe3a4d841c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 06:27:28.206326 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:28.201898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33670 (local address 127.15.254.61:39745)
0807 06:27:28.202194 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:28.202199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:28.202376 (+ 177us) server_negotiation.cc:407] Connection header received
0807 06:27:28.202581 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:28.202583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:28.202720 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:28.202849 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:27:28.203198 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:28.203778 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:28.204710 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:28.204962 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:28.205085 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:28.205438 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:27:28.205900 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:27:28.206051 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:28.206093 (+ 42us) server_negotiation.cc:299] Negotiation successful
0807 06:27:28.206152 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":120,"threads_started":1}
I20260807 06:27:28.214524 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:28.210305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.15.254.60:46459)
0807 06:27:28.210605 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:28.210609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:28.210623 (+ 14us) server_negotiation.cc:407] Connection header received
0807 06:27:28.210711 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:28.210713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:28.210831 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:28.210946 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:27:28.211374 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:28.211898 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:28.212835 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:28.213074 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:28.213160 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:28.213534 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:27:28.214015 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:27:28.214108 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:28.214238 (+ 130us) server_negotiation.cc:299] Negotiation successful
0807 06:27:28.214317 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":115,"threads_started":1}
I20260807 06:27:28.222546 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:28.217811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.15.254.61:39745)
0807 06:27:28.217955 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:28.217959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:28.218392 (+ 433us) server_negotiation.cc:407] Connection header received
0807 06:27:28.218587 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:28.218590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:28.218643 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:28.218734 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:27:28.218977 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:28.219801 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:28.221405 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:28.221657 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:28.221779 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:28.222046 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:27:28.222126 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:27:28.222212 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:28.222349 (+ 137us) server_negotiation.cc:299] Negotiation successful
0807 06:27:28.222417 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 06:27:28.222935 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:28.218060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35932 (local address 127.15.254.60:46459)
0807 06:27:28.218184 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:28.218187 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:28.219091 (+ 904us) server_negotiation.cc:407] Connection header received
0807 06:27:28.219178 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:28.219184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:28.219240 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:28.219338 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:27:28.219602 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:28.220200 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:28.221375 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:28.221724 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:28.221822 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:28.222357 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:27:28.222437 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:27:28.222554 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:28.222709 (+ 155us) server_negotiation.cc:299] Negotiation successful
0807 06:27:28.222768 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260807 06:27:28.223421 16551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:27:28.218209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35782 (local address 127.15.254.62:43019)
0807 06:27:28.218521 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0807 06:27:28.218525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:27:28.218540 (+ 15us) server_negotiation.cc:407] Connection header received
0807 06:27:28.218623 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:27:28.218626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:27:28.218740 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:27:28.218892 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:27:28.219242 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:28.220014 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:28.221285 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:27:28.221688 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:27:28.221808 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:27:28.222375 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:27:28.222720 (+ 345us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:27:28.222813 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 06:27:28.223217 (+ 404us) server_negotiation.cc:299] Negotiation successful
0807 06:27:28.223277 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":75,"threads_started":1}
I20260807 06:27:28.230890 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 16016
I20260807 06:27:28.244746 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 16119
I20260807 06:27:28.255560 16376 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZSVWny/build/debug/bin/kudu with pid 16229
2026-08-07T06:27:28Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3367 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6692 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (309291 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST